Jul 22 09:17:48.301700 ip-10-0-138-138 systemd[1]: Starting Kubernetes Kubelet... Jul 22 09:17:48.616026 ip-10-0-138-138 kubenswrapper[2549]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 09:17:48.616026 ip-10-0-138-138 kubenswrapper[2549]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 09:17:48.616026 ip-10-0-138-138 kubenswrapper[2549]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 09:17:48.616026 ip-10-0-138-138 kubenswrapper[2549]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 09:17:48.616026 ip-10-0-138-138 kubenswrapper[2549]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 09:17:48.616026 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.606886 2549 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 09:17:48.616026 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613618 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 09:17:48.616026 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613630 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 09:17:48.616026 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613634 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 09:17:48.616026 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613636 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 09:17:48.616026 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613639 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 09:17:48.616026 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613642 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 09:17:48.616026 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613645 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 09:17:48.616026 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613648 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 09:17:48.621078 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613650 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 09:17:48.621078 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613653 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 09:17:48.621078 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613655 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 09:17:48.621078 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613658 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 09:17:48.621078 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613661 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 09:17:48.621078 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613663 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 09:17:48.621078 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613666 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 09:17:48.621078 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613668 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 09:17:48.621078 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613670 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 09:17:48.621078 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613673 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 09:17:48.621078 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613676 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 09:17:48.621078 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613680 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 09:17:48.621078 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613682 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 09:17:48.621078 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613691 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 09:17:48.621078 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613696 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 09:17:48.621078 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613699 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 09:17:48.621078 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613702 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 09:17:48.621078 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613705 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 09:17:48.621078 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613708 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 09:17:48.621078 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613710 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 09:17:48.622113 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613713 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 09:17:48.622113 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613716 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 09:17:48.622113 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613719 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 09:17:48.622113 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613721 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 09:17:48.622113 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613724 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 09:17:48.622113 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613726 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 09:17:48.622113 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613729 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 09:17:48.622113 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613731 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 09:17:48.622113 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613734 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 09:17:48.622113 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613736 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 09:17:48.622113 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613739 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 09:17:48.622113 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613742 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 09:17:48.622113 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613744 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 09:17:48.622113 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613747 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 09:17:48.622113 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613749 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 09:17:48.622113 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613752 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 09:17:48.622113 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613754 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 09:17:48.622113 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613756 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 09:17:48.622113 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613759 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 09:17:48.622113 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613761 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 09:17:48.625828 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613764 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 09:17:48.625828 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613767 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 09:17:48.625828 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613770 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 09:17:48.625828 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613772 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 09:17:48.625828 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613781 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 09:17:48.625828 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613785 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 09:17:48.625828 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613788 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 09:17:48.625828 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613790 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 09:17:48.625828 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613793 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 09:17:48.625828 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613796 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 09:17:48.625828 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613798 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 09:17:48.625828 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613801 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 09:17:48.625828 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613803 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 09:17:48.625828 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613807 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 09:17:48.625828 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613810 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 09:17:48.625828 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613813 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 09:17:48.625828 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613815 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 09:17:48.625828 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613818 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 09:17:48.625828 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613820 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 09:17:48.625828 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613823 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 09:17:48.626847 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613826 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 09:17:48.626847 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613828 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 09:17:48.626847 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613831 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 09:17:48.626847 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613833 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 09:17:48.626847 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613836 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 09:17:48.626847 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613838 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 09:17:48.626847 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613841 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 09:17:48.626847 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613843 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 09:17:48.626847 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613846 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 09:17:48.626847 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613848 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 09:17:48.626847 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613851 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 09:17:48.626847 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613853 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 09:17:48.626847 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613856 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 09:17:48.626847 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613858 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 09:17:48.626847 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613861 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 09:17:48.626847 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613864 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 09:17:48.626847 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613866 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 09:17:48.626847 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613869 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 09:17:48.626847 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613872 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 09:17:48.626847 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613875 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 09:17:48.630126 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613880 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 09:17:48.630126 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613883 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 09:17:48.630126 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613886 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 09:17:48.630126 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613889 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 09:17:48.630126 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613892 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 09:17:48.630126 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613894 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 09:17:48.630126 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613897 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 09:17:48.630126 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613900 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 22 09:17:48.630126 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613902 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 09:17:48.630126 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613904 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 09:17:48.630126 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.613907 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 09:17:48.630126 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614604 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 09:17:48.630126 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614611 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 09:17:48.630126 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614614 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 09:17:48.630126 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614616 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 09:17:48.630126 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614619 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 09:17:48.630126 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614622 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 09:17:48.630126 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614624 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 09:17:48.630126 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614627 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 09:17:48.630126 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614630 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 09:17:48.631135 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614632 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 09:17:48.631135 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614634 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 09:17:48.631135 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614637 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 09:17:48.631135 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614640 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 09:17:48.631135 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614642 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 09:17:48.631135 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614645 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 09:17:48.631135 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614648 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 09:17:48.631135 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614650 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 09:17:48.631135 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614653 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 09:17:48.631135 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614655 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 09:17:48.631135 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614658 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 09:17:48.631135 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614660 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 09:17:48.631135 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614664 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 09:17:48.631135 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614666 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 09:17:48.631135 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614669 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 09:17:48.631135 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614672 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 09:17:48.631135 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614674 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 09:17:48.631135 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614677 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 09:17:48.631135 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614680 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 09:17:48.631135 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614682 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 09:17:48.636245 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614685 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 09:17:48.636245 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614688 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 09:17:48.636245 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614691 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 09:17:48.636245 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614694 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 09:17:48.636245 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614696 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 09:17:48.636245 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614699 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 09:17:48.636245 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614702 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 09:17:48.636245 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614706 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 09:17:48.636245 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614709 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 09:17:48.636245 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614712 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 09:17:48.636245 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614715 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 09:17:48.636245 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614717 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 09:17:48.636245 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614720 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 09:17:48.636245 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614722 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 09:17:48.636245 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614724 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 09:17:48.636245 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614727 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 09:17:48.636245 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614729 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 09:17:48.636245 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614733 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 09:17:48.636245 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614735 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 09:17:48.636245 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614738 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 09:17:48.637293 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614740 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 09:17:48.637293 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614743 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 09:17:48.637293 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614746 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 09:17:48.637293 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614748 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 09:17:48.637293 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614750 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 09:17:48.637293 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614753 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 09:17:48.637293 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614756 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 09:17:48.637293 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614758 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 09:17:48.637293 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614761 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 09:17:48.637293 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614763 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 09:17:48.637293 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614765 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 09:17:48.637293 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614768 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 09:17:48.637293 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614770 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 09:17:48.637293 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614772 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 09:17:48.637293 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614777 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 09:17:48.637293 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614780 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 09:17:48.637293 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614783 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 09:17:48.637293 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614786 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 09:17:48.637293 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614790 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 09:17:48.641968 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614794 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 09:17:48.641968 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614797 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 09:17:48.641968 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614800 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 09:17:48.641968 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614803 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 09:17:48.641968 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614805 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 09:17:48.641968 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614808 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 09:17:48.641968 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614811 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 09:17:48.641968 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614814 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 09:17:48.641968 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614816 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 09:17:48.641968 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614819 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 09:17:48.641968 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614821 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 09:17:48.641968 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614824 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 09:17:48.641968 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614827 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 09:17:48.641968 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614829 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 09:17:48.641968 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614831 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 09:17:48.641968 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614834 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 09:17:48.641968 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614837 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 09:17:48.641968 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614839 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 09:17:48.641968 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614841 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 09:17:48.641968 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614844 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 09:17:48.642975 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614846 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 09:17:48.642975 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614849 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 09:17:48.642975 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614851 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 09:17:48.642975 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614854 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 22 09:17:48.642975 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614856 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 09:17:48.642975 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614859 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 09:17:48.642975 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614861 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 09:17:48.642975 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614864 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 09:17:48.642975 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614866 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 09:17:48.642975 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614869 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 09:17:48.642975 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.614872 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 09:17:48.642975 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616245 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 09:17:48.642975 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616251 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 09:17:48.642975 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616255 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 09:17:48.642975 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616258 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 09:17:48.642975 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616260 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 09:17:48.642975 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616265 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 09:17:48.642975 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616269 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 09:17:48.642975 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616272 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 09:17:48.646708 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616274 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 09:17:48.646708 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616278 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 09:17:48.646708 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616281 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 09:17:48.646708 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616284 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 09:17:48.646708 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616287 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 09:17:48.646708 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616290 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 09:17:48.646708 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616292 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 09:17:48.646708 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616295 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 09:17:48.646708 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616298 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 09:17:48.646708 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616300 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 09:17:48.646708 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616303 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 09:17:48.646708 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616305 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 09:17:48.646708 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616309 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 09:17:48.646708 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616311 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 09:17:48.646708 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616314 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 09:17:48.646708 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616317 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 09:17:48.646708 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616319 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 09:17:48.646708 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616322 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 09:17:48.646708 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616324 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 09:17:48.646708 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616327 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 09:17:48.647733 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616330 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 22 09:17:48.647733 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616332 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 09:17:48.647733 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616335 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 09:17:48.647733 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616337 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 09:17:48.647733 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616340 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 09:17:48.647733 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616343 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 09:17:48.647733 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616345 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 09:17:48.647733 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616348 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 09:17:48.647733 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616350 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 09:17:48.647733 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616353 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 09:17:48.647733 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616355 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 09:17:48.647733 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616357 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 09:17:48.647733 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616360 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 09:17:48.647733 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616362 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 09:17:48.647733 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616365 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 09:17:48.647733 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616368 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 09:17:48.647733 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616387 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 09:17:48.647733 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616392 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 09:17:48.647733 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616395 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 09:17:48.647733 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616398 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 09:17:48.650966 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616401 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 09:17:48.650966 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616403 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 09:17:48.650966 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616406 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 09:17:48.650966 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616408 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 09:17:48.650966 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616412 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 09:17:48.650966 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616416 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 09:17:48.650966 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616418 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 09:17:48.650966 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616421 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 09:17:48.650966 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616423 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 09:17:48.650966 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616426 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 09:17:48.650966 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616429 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 09:17:48.650966 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616432 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 09:17:48.650966 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616436 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 09:17:48.650966 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616439 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 09:17:48.650966 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616444 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 09:17:48.650966 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616447 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 09:17:48.650966 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616450 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 09:17:48.650966 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616453 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 09:17:48.650966 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616456 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 09:17:48.650966 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616458 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 09:17:48.651997 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616461 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 09:17:48.651997 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616466 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 09:17:48.651997 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616468 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 09:17:48.651997 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616471 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 09:17:48.651997 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616474 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 09:17:48.651997 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616476 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 09:17:48.651997 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616480 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 09:17:48.651997 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616482 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 09:17:48.651997 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616485 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 09:17:48.651997 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616487 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 09:17:48.651997 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616490 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 09:17:48.651997 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616492 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 09:17:48.651997 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616495 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 09:17:48.651997 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616497 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 09:17:48.651997 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616500 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 09:17:48.651997 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616503 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 09:17:48.651997 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616505 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 09:17:48.651997 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616508 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 09:17:48.651997 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616511 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 09:17:48.651997 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616513 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 09:17:48.753192 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616516 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 09:17:48.753192 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616518 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 09:17:48.753192 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616521 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 09:17:48.753192 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616524 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 09:17:48.753192 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616527 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 09:17:48.753192 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616529 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 09:17:48.753192 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616532 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 09:17:48.753192 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616534 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 09:17:48.753192 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616537 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 09:17:48.753192 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616540 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 09:17:48.753192 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.616542 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 09:17:48.753192 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616612 2549 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 09:17:48.753192 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616619 2549 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 09:17:48.753192 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616628 2549 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 09:17:48.753192 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616634 2549 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 09:17:48.753192 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616640 2549 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 09:17:48.753192 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616644 2549 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 09:17:48.753192 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616648 2549 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 09:17:48.753192 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616652 2549 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 09:17:48.753192 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616655 2549 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 09:17:48.753192 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616658 2549 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616661 2549 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616665 2549 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616668 2549 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616671 2549 flags.go:64] FLAG: --cgroup-root="" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616674 2549 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616676 2549 flags.go:64] FLAG: --client-ca-file="" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616679 2549 flags.go:64] FLAG: --cloud-provider="external" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616682 2549 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616685 2549 flags.go:64] FLAG: --cluster-domain="" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616688 2549 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616692 2549 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616695 2549 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616698 2549 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616702 2549 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616705 2549 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616708 2549 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616711 2549 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616714 2549 flags.go:64] FLAG: --contention-profiling="false" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616718 2549 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616720 2549 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616724 2549 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616726 2549 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616730 2549 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 09:17:48.754187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616734 2549 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616738 2549 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616740 2549 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616743 2549 flags.go:64] FLAG: --enable-server="true" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616746 2549 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616750 2549 flags.go:64] FLAG: --event-burst="100" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616753 2549 flags.go:64] FLAG: --event-qps="50" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616756 2549 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616759 2549 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616762 2549 flags.go:64] FLAG: --eviction-hard="" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616765 2549 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616768 2549 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616771 2549 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616774 2549 flags.go:64] FLAG: --eviction-soft="" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616776 2549 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616779 2549 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616782 2549 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616785 2549 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616788 2549 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616791 2549 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616795 2549 flags.go:64] FLAG: --feature-gates="" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616798 2549 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616801 2549 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616804 2549 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616807 2549 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616810 2549 flags.go:64] FLAG: --healthz-port="10248" Jul 22 09:17:48.759693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616813 2549 flags.go:64] FLAG: --help="false" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616816 2549 flags.go:64] FLAG: --hostname-override="ip-10-0-138-138.ec2.internal" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616819 2549 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616822 2549 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616825 2549 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616828 2549 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616832 2549 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616835 2549 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616838 2549 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616841 2549 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616844 2549 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616847 2549 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616850 2549 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616853 2549 flags.go:64] FLAG: --kube-reserved="" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616855 2549 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616858 2549 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616861 2549 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616864 2549 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616866 2549 flags.go:64] FLAG: --lock-file="" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616869 2549 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616872 2549 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616875 2549 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616880 2549 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 09:17:48.761170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616883 2549 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616886 2549 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616889 2549 flags.go:64] FLAG: --logging-format="text" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616892 2549 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616896 2549 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616898 2549 flags.go:64] FLAG: --manifest-url="" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616901 2549 flags.go:64] FLAG: --manifest-url-header="" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616905 2549 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616908 2549 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616912 2549 flags.go:64] FLAG: --max-pods="110" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616929 2549 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616933 2549 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616936 2549 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616940 2549 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616943 2549 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616946 2549 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616950 2549 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616956 2549 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616960 2549 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616963 2549 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616966 2549 flags.go:64] FLAG: --pod-cidr="" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616969 2549 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616974 2549 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616977 2549 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 09:17:48.762189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616980 2549 flags.go:64] FLAG: --pods-per-core="0" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616983 2549 flags.go:64] FLAG: --port="10250" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616986 2549 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616989 2549 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a25d3ee2070f0a1c" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616992 2549 flags.go:64] FLAG: --qos-reserved="" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616995 2549 flags.go:64] FLAG: --read-only-port="10255" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.616998 2549 flags.go:64] FLAG: --register-node="true" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617001 2549 flags.go:64] FLAG: --register-with-taints="" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617004 2549 flags.go:64] FLAG: --registry-burst="10" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617007 2549 flags.go:64] FLAG: --registry-qps="5" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617014 2549 flags.go:64] FLAG: --reserved-cpus="" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617017 2549 flags.go:64] FLAG: --reserved-memory="" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617029 2549 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617032 2549 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617035 2549 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617039 2549 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617041 2549 flags.go:64] FLAG: --runonce="false" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617044 2549 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617047 2549 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617050 2549 flags.go:64] FLAG: --seccomp-default="false" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617053 2549 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617056 2549 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617058 2549 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617061 2549 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617064 2549 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617068 2549 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 09:17:48.767266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617071 2549 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617074 2549 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617077 2549 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617080 2549 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617083 2549 flags.go:64] FLAG: --system-cgroups="" Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617086 2549 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617091 2549 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617094 2549 flags.go:64] FLAG: --tls-cert-file="" Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617096 2549 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617100 2549 flags.go:64] FLAG: --tls-min-version="" Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617103 2549 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617105 2549 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617108 2549 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617111 2549 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617113 2549 flags.go:64] FLAG: --v="2" Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617117 2549 flags.go:64] FLAG: --version="false" Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617122 2549 flags.go:64] FLAG: --vmodule="" Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617126 2549 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617129 2549 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617279 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617286 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617288 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617291 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617294 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 09:17:48.768339 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617297 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 09:17:48.769393 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617300 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 09:17:48.769393 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617302 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 09:17:48.769393 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617305 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 09:17:48.769393 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617307 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 09:17:48.769393 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617310 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 09:17:48.769393 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617313 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 09:17:48.769393 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617317 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 09:17:48.769393 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617320 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 09:17:48.769393 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617323 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 09:17:48.769393 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617325 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 09:17:48.769393 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617328 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 09:17:48.769393 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617331 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 09:17:48.769393 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617333 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 09:17:48.769393 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617336 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 09:17:48.769393 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617339 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 09:17:48.769393 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617341 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 09:17:48.769393 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617344 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 09:17:48.769393 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617346 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 09:17:48.769393 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617349 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 09:17:48.769393 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617351 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 09:17:48.772220 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617354 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 09:17:48.772220 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617356 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 09:17:48.772220 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617359 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 09:17:48.772220 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617363 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 09:17:48.772220 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617365 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 09:17:48.772220 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617368 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 09:17:48.772220 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617371 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 09:17:48.772220 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617389 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 09:17:48.772220 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617398 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 09:17:48.772220 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617401 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 09:17:48.772220 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617405 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 09:17:48.772220 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617408 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 09:17:48.772220 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617412 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 09:17:48.772220 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617422 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 09:17:48.772220 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617426 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 09:17:48.772220 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617428 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 09:17:48.772220 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617431 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 09:17:48.772220 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617434 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 09:17:48.772220 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617437 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 09:17:48.773221 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617440 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 09:17:48.773221 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617442 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 09:17:48.773221 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617445 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 09:17:48.773221 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617448 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 09:17:48.773221 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617450 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 09:17:48.773221 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617453 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 09:17:48.773221 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617455 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 09:17:48.773221 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617458 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 09:17:48.773221 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617461 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 09:17:48.773221 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617463 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 09:17:48.773221 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617465 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 09:17:48.773221 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617468 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 09:17:48.773221 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617470 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 09:17:48.773221 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617474 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 09:17:48.773221 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617477 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 09:17:48.773221 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617480 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 09:17:48.773221 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617485 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 09:17:48.773221 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617488 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 09:17:48.773221 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617490 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 09:17:48.773221 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617493 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 09:17:48.776939 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617495 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 09:17:48.776939 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617498 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 09:17:48.776939 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617506 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 09:17:48.776939 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617509 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 09:17:48.776939 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617512 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 09:17:48.776939 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617514 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 09:17:48.776939 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617517 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 09:17:48.776939 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617519 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 09:17:48.776939 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617522 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 09:17:48.776939 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617525 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 09:17:48.776939 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617529 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 09:17:48.776939 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617533 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 09:17:48.776939 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617536 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 09:17:48.776939 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617538 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 09:17:48.776939 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617541 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 09:17:48.776939 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617545 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 09:17:48.776939 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617547 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 09:17:48.776939 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617550 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 09:17:48.776939 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617553 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 09:17:48.777919 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617555 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 09:17:48.777919 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617558 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 09:17:48.777919 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617560 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 09:17:48.777919 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617563 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 09:17:48.777919 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617565 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 09:17:48.777919 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617568 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 22 09:17:48.777919 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617570 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 09:17:48.777919 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617572 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 09:17:48.777919 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617575 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 09:17:48.777919 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617579 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 09:17:48.777919 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617581 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 09:17:48.777919 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617584 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 09:17:48.777919 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617586 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 09:17:48.777919 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617589 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 09:17:48.777919 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.617591 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 09:17:50.013754 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.617597 2549 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 09:17:50.013754 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.625130 2549 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 09:17:50.013754 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.625145 2549 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 09:17:50.013754 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.625158 2549 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 09:17:50.013754 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.625167 2549 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 09:17:50.013754 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625238 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 09:17:50.013754 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625242 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 09:17:50.013754 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625246 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 22 09:17:50.013754 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625249 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 09:17:50.013754 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625251 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 09:17:50.013754 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625255 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 09:17:50.013754 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625258 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 09:17:50.013754 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625260 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 09:17:50.013754 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625263 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 09:17:50.014610 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625266 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 09:17:50.014610 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625269 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 09:17:50.014610 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625272 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 09:17:50.014610 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625274 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 09:17:50.014610 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625277 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 09:17:50.014610 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625279 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 09:17:50.014610 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625282 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 09:17:50.014610 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625289 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 09:17:50.014610 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625291 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 09:17:50.014610 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625296 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 09:17:50.014610 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625299 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 09:17:50.014610 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625302 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 09:17:50.014610 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625306 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 09:17:50.014610 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625308 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 09:17:50.014610 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625311 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 09:17:50.014610 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625314 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 09:17:50.014610 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625316 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 09:17:50.014610 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625319 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 09:17:50.014610 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625321 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 09:17:50.128916 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625324 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 09:17:50.128916 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625327 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 09:17:50.128916 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625329 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 09:17:50.128916 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625332 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 09:17:50.128916 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625334 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 09:17:50.128916 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625337 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 09:17:50.128916 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625340 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 09:17:50.128916 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625342 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 09:17:50.128916 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625345 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 09:17:50.128916 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625347 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 09:17:50.128916 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625350 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 09:17:50.128916 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625352 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 09:17:50.128916 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625355 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 09:17:50.128916 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625358 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 09:17:50.128916 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625360 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 09:17:50.128916 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625363 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 09:17:50.128916 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625365 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 09:17:50.128916 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625369 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 09:17:50.128916 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625388 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 09:17:50.130223 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625393 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 09:17:50.130223 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625395 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 09:17:50.130223 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625398 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 09:17:50.130223 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625400 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 09:17:50.130223 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625407 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 09:17:50.130223 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625409 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 09:17:50.130223 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625412 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 09:17:50.130223 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625424 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 09:17:50.130223 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625427 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 09:17:50.130223 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625429 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 09:17:50.130223 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625432 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 09:17:50.130223 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625434 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 09:17:50.130223 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625437 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 09:17:50.130223 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625440 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 09:17:50.130223 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625442 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 09:17:50.130223 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625446 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 09:17:50.130223 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625450 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 09:17:50.130223 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625453 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 09:17:50.130223 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625456 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 09:17:50.130223 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625459 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 09:17:50.131107 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625462 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 09:17:50.131107 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625464 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 09:17:50.131107 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625467 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 09:17:50.131107 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625469 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 09:17:50.131107 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625472 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 09:17:50.131107 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625474 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 09:17:50.131107 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625477 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 09:17:50.131107 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625479 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 09:17:50.131107 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625482 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 09:17:50.131107 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625485 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 09:17:50.131107 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625487 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 09:17:50.131107 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625490 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 09:17:50.131107 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625493 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 09:17:50.131107 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625496 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 09:17:50.131107 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625498 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 09:17:50.131107 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625501 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 09:17:50.131107 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625505 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 09:17:50.131107 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625507 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 09:17:50.131107 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625510 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 09:17:50.131107 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625512 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 09:17:50.131107 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625515 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 09:17:50.132833 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625518 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 09:17:50.132833 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625520 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 09:17:50.132833 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625523 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 09:17:50.132833 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625525 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 09:17:50.132833 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625528 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 09:17:50.132833 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625530 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 09:17:50.132833 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625533 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 09:17:50.132833 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625535 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 09:17:50.132833 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625538 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 09:17:50.132833 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625540 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 09:17:50.132833 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625543 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 09:17:50.132833 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.625548 2549 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 09:17:50.132833 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625675 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 09:17:50.132833 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625681 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 09:17:50.133281 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625684 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 09:17:50.133281 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625687 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 09:17:50.133281 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625690 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 09:17:50.133281 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625693 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 09:17:50.133281 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625696 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 09:17:50.133281 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625698 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 09:17:50.133281 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625701 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 09:17:50.133281 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625704 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 09:17:50.133281 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625706 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 09:17:50.133281 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625708 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 09:17:50.133281 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625712 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 09:17:50.133281 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625715 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 09:17:50.133281 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625717 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 09:17:50.133281 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625721 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 09:17:50.133281 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625724 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 09:17:50.133281 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625726 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 09:17:50.133281 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625729 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 09:17:50.133281 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625731 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 09:17:50.133281 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625734 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 09:17:50.133281 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625736 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 09:17:50.135016 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625739 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 09:17:50.135016 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625741 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 09:17:50.135016 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625744 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 09:17:50.135016 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625746 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 09:17:50.135016 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625751 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 09:17:50.135016 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625753 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 09:17:50.135016 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625756 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 09:17:50.135016 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625758 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 09:17:50.135016 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625761 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 09:17:50.135016 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625763 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 09:17:50.135016 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625766 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 09:17:50.135016 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625768 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 09:17:50.135016 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625771 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 09:17:50.135016 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625773 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 09:17:50.135016 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625776 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 09:17:50.135016 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625778 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 09:17:50.135016 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625780 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 09:17:50.135016 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625783 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 09:17:50.135016 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625785 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 09:17:50.135016 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625788 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 09:17:50.135865 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625790 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 09:17:50.135865 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625793 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 09:17:50.135865 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625795 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 09:17:50.135865 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625798 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 09:17:50.135865 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625801 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 09:17:50.135865 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625805 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 09:17:50.135865 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625807 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 09:17:50.135865 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625810 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 09:17:50.135865 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625812 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 09:17:50.135865 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625816 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 09:17:50.135865 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625820 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 09:17:50.135865 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625823 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 09:17:50.135865 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625825 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 09:17:50.135865 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625828 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 09:17:50.135865 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625831 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 09:17:50.135865 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625834 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 09:17:50.135865 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625838 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 09:17:50.135865 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625840 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 09:17:50.135865 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625843 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 09:17:50.135865 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625846 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 09:17:50.137148 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625849 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 09:17:50.137148 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625852 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 09:17:50.137148 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625855 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 09:17:50.137148 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625858 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 09:17:50.137148 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625861 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 09:17:50.137148 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625864 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 09:17:50.137148 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625867 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 09:17:50.137148 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625869 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 22 09:17:50.137148 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625871 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 09:17:50.137148 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625874 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 09:17:50.137148 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625877 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 09:17:50.137148 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625879 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 09:17:50.137148 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625882 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 09:17:50.137148 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625884 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 09:17:50.137148 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625887 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 09:17:50.137148 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625889 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 09:17:50.137148 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625893 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 09:17:50.137148 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625896 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 09:17:50.137148 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625899 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 09:17:50.139805 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625902 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 09:17:50.139805 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625904 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 09:17:50.139805 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625906 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 09:17:50.139805 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625909 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 09:17:50.139805 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625911 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 09:17:50.139805 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625914 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 09:17:50.139805 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625918 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 09:17:50.139805 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625921 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 09:17:50.139805 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625924 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 09:17:50.139805 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625929 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 09:17:50.139805 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625932 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 09:17:50.139805 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625935 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 09:17:50.139805 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625937 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 09:17:50.139805 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625940 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 09:17:50.139805 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625943 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 09:17:50.139805 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625945 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 09:17:50.139805 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625948 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 09:17:50.139805 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:48.625950 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 09:17:50.140316 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.625955 2549 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 09:17:50.140316 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.626065 2549 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 09:17:50.140316 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.628135 2549 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 09:17:50.140316 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.629443 2549 server.go:1019] "Starting client certificate rotation" Jul 22 09:17:50.140316 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.629539 2549 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 09:17:50.140316 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.629565 2549 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 09:17:50.140316 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.631615 2549 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 09:17:50.140316 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.633135 2549 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 09:17:50.140316 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.643358 2549 log.go:25] "Validated CRI v1 runtime API" Jul 22 09:17:50.140316 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.647574 2549 log.go:25] "Validated CRI v1 image API" Jul 22 09:17:50.140316 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.648389 2549 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 09:17:50.140316 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.650576 2549 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7d8d0742-30c9-4e82-af48-48a5c193b38c:/dev/nvme0n1p3 9f3fa8b1-74d7-47f7-92c7-e3159dab5313:/dev/nvme0n1p4] Jul 22 09:17:50.140636 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.650591 2549 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 09:17:50.140666 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.656574 2549 manager.go:216] Machine: {Timestamp:2026-07-22 09:17:48.655291672 +0000 UTC m=+0.254013403 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100136 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d6c8d41586ab4c214ffa0d240cd97 SystemUUID:ec2d6c8d-4158-6ab4-c214-ffa0d240cd97 BootID:248bb437-b9e8-41bc-bc1f-bab7b5fa9d4a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0e:9b:b0:b1:9d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0e:9b:b0:b1:9d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:ca:e5:59:a2:c5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 09:17:50.140666 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.656671 2549 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 09:17:50.140666 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.656752 2549 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 09:17:50.140666 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.658927 2549 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 09:17:50.140666 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.659112 2549 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 09:17:50.140666 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.659137 2549 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-138.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 09:17:50.140666 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.659703 2549 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.659714 2549 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.659722 2549 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.659735 2549 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.660831 2549 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.664351 2549 state_mem.go:36] "Initialized new in-memory state store" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.664477 2549 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.664505 2549 kubelet.go:486] "Attempting to sync node with API server" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.664514 2549 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.664527 2549 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.664538 2549 kubelet.go:398] "Adding apiserver pod source" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.664546 2549 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.665016 2549 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.665032 2549 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.666713 2549 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.668562 2549 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.668589 2549 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.669619 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.669636 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.669646 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.669653 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.669663 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.669669 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.669678 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.669685 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.669694 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.669699 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.669708 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.669713 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.670742 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.670753 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.679549 2549 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-138.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:48.679695 2549 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-138.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:48.679696 2549 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.691410 2549 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nb6jj" Jul 22 09:17:50.140877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:48.699592 2549 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nb6jj" Jul 22 09:17:50.196921 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.196898 2549 server.go:1268] "Started kubelet" Jul 22 09:17:50.197023 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.196964 2549 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 09:17:50.197023 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.196974 2549 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 09:17:50.197123 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.197028 2549 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 09:17:50.197754 ip-10-0-138-138 systemd[1]: Started Kubernetes Kubelet. Jul 22 09:17:50.198236 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.198186 2549 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 09:17:50.198899 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.198884 2549 server.go:310] "Adding debug handlers to kubelet server" Jul 22 09:17:50.201355 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:50.201332 2549 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 09:17:50.205277 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.205259 2549 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 09:17:50.205277 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.205269 2549 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 09:17:50.205429 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:50.205385 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-138.ec2.internal\" not found" Jul 22 09:17:50.205832 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.205813 2549 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 09:17:50.205832 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.205829 2549 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 09:17:50.205947 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.205890 2549 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 09:17:50.206082 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.206062 2549 reconstruct.go:97] "Volume reconstruction finished" Jul 22 09:17:50.206082 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.206080 2549 reconciler.go:29] "Reconciler: start to sync state" Jul 22 09:17:50.206200 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.206171 2549 factory.go:55] Registering systemd factory Jul 22 09:17:50.206247 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.206206 2549 factory.go:223] Registration of the systemd container factory successfully Jul 22 09:17:50.206448 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.206435 2549 factory.go:153] Registering CRI-O factory Jul 22 09:17:50.206448 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.206450 2549 factory.go:223] Registration of the crio container factory successfully Jul 22 09:17:50.206533 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.206504 2549 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 09:17:50.206533 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.206527 2549 factory.go:103] Registering Raw factory Jul 22 09:17:50.206613 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.206540 2549 manager.go:1196] Started watching for new ooms in manager Jul 22 09:17:50.206996 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.206981 2549 manager.go:349] Starting recovery of all containers Jul 22 09:17:50.210823 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.210798 2549 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 09:17:50.219353 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.219331 2549 manager.go:354] Recovery completed Jul 22 09:17:50.219679 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:50.219647 2549 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-138.ec2.internal\" not found" node="ip-10-0-138-138.ec2.internal" Jul 22 09:17:50.223374 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.223361 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 09:17:50.226806 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.226790 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientMemory" Jul 22 09:17:50.226882 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.226823 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 09:17:50.226882 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.226836 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientPID" Jul 22 09:17:50.227405 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.227386 2549 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 09:17:50.227405 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.227404 2549 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 09:17:50.227554 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.227413 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 09:17:50.227554 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.227423 2549 state_mem.go:36] "Initialized new in-memory state store" Jul 22 09:17:50.228288 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.228269 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientMemory" Jul 22 09:17:50.228384 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.228295 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 09:17:50.228384 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.228333 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientPID" Jul 22 09:17:50.229623 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.229612 2549 policy_none.go:49] "None policy: Start" Jul 22 09:17:50.229668 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.229627 2549 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 09:17:50.229668 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.229637 2549 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 09:17:50.231597 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.231586 2549 policy_none.go:47] "Start" Jul 22 09:17:50.266863 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.266850 2549 manager.go:337] "Starting Device Plugin manager" Jul 22 09:17:50.266938 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:50.266880 2549 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 09:17:50.266938 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.266890 2549 server.go:87] "Starting device plugin registration server" Jul 22 09:17:50.267167 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.267153 2549 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 09:17:50.267254 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.267167 2549 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 09:17:50.267319 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.267285 2549 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 09:17:50.267416 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.267382 2549 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 09:17:50.267416 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.267403 2549 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 09:17:50.267945 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:50.267902 2549 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 09:17:50.268026 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:50.268013 2549 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-138.ec2.internal\" not found" Jul 22 09:17:50.332069 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.332019 2549 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 09:17:50.334233 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.333207 2549 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 09:17:50.334233 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.333242 2549 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 09:17:50.334233 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.333270 2549 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 09:17:50.334233 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:50.333316 2549 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 09:17:50.336284 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.336269 2549 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 09:17:50.367895 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.367879 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 09:17:50.369238 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.369221 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientMemory" Jul 22 09:17:50.369338 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.369253 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 09:17:50.369338 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.369263 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientPID" Jul 22 09:17:50.369338 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.369284 2549 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-138.ec2.internal" Jul 22 09:17:50.385696 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.385542 2549 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-138.ec2.internal" Jul 22 09:17:50.385746 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:50.385704 2549 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-138.ec2.internal\": node \"ip-10-0-138-138.ec2.internal\" not found" Jul 22 09:17:50.432496 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:50.432440 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-138.ec2.internal\" not found" Jul 22 09:17:50.433402 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.433376 2549 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-138.ec2.internal"] Jul 22 09:17:50.433469 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.433460 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 09:17:50.434279 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.434265 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientMemory" Jul 22 09:17:50.434350 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.434296 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 09:17:50.434350 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.434328 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientPID" Jul 22 09:17:50.436699 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.436687 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 09:17:50.436834 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.436816 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" Jul 22 09:17:50.436870 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.436856 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 09:17:50.437347 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.437333 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientMemory" Jul 22 09:17:50.437409 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.437368 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 09:17:50.437409 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.437378 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientPID" Jul 22 09:17:50.438169 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.438149 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientMemory" Jul 22 09:17:50.438169 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.438170 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 09:17:50.438279 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.438181 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientPID" Jul 22 09:17:50.440424 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.440409 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-138.ec2.internal" Jul 22 09:17:50.440493 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.440436 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 09:17:50.441033 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.441017 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientMemory" Jul 22 09:17:50.441100 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.441046 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 09:17:50.441100 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.441060 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeHasSufficientPID" Jul 22 09:17:50.462888 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:50.462867 2549 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-138.ec2.internal\" not found" node="ip-10-0-138-138.ec2.internal" Jul 22 09:17:50.467242 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:50.467228 2549 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-138.ec2.internal\" not found" node="ip-10-0-138-138.ec2.internal" Jul 22 09:17:50.508293 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.508266 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/16c453564e70f602f26f88ff0da5d012-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal\" (UID: \"16c453564e70f602f26f88ff0da5d012\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" Jul 22 09:17:50.508416 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.508298 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16c453564e70f602f26f88ff0da5d012-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal\" (UID: \"16c453564e70f602f26f88ff0da5d012\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" Jul 22 09:17:50.508416 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.508388 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/22c71e61550bcbc1af7cc7ad68e31b2b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-138.ec2.internal\" (UID: \"22c71e61550bcbc1af7cc7ad68e31b2b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-138.ec2.internal" Jul 22 09:17:50.532740 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:50.532722 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-138.ec2.internal\" not found" Jul 22 09:17:50.609165 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.609137 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/16c453564e70f602f26f88ff0da5d012-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal\" (UID: \"16c453564e70f602f26f88ff0da5d012\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" Jul 22 09:17:50.609269 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.609171 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16c453564e70f602f26f88ff0da5d012-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal\" (UID: \"16c453564e70f602f26f88ff0da5d012\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" Jul 22 09:17:50.609269 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.609191 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/22c71e61550bcbc1af7cc7ad68e31b2b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-138.ec2.internal\" (UID: \"22c71e61550bcbc1af7cc7ad68e31b2b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-138.ec2.internal" Jul 22 09:17:50.609269 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.609241 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/22c71e61550bcbc1af7cc7ad68e31b2b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-138.ec2.internal\" (UID: \"22c71e61550bcbc1af7cc7ad68e31b2b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-138.ec2.internal" Jul 22 09:17:50.609269 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.609240 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16c453564e70f602f26f88ff0da5d012-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal\" (UID: \"16c453564e70f602f26f88ff0da5d012\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" Jul 22 09:17:50.609432 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.609240 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/16c453564e70f602f26f88ff0da5d012-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal\" (UID: \"16c453564e70f602f26f88ff0da5d012\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" Jul 22 09:17:50.633384 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:50.633354 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-138.ec2.internal\" not found" Jul 22 09:17:50.733908 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:50.733842 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-138.ec2.internal\" not found" Jul 22 09:17:50.764002 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.763973 2549 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 09:17:50.764394 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.764128 2549 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 09:17:50.764394 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.764165 2549 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 09:17:50.766149 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.766132 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" Jul 22 09:17:50.769727 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.769684 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-138.ec2.internal" Jul 22 09:17:50.813679 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.813652 2549 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 09:17:50.834154 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:50.834125 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-138.ec2.internal\" not found" Jul 22 09:17:50.839289 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.839254 2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 09:12:49 +0000 UTC" deadline="2028-02-07 17:33:54.727891861 +0000 UTC" Jul 22 09:17:50.839349 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:50.839289 2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13568h16m3.888604853s" Jul 22 09:17:50.934770 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:50.934738 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-138.ec2.internal\" not found" Jul 22 09:17:51.035330 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:51.035233 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-138.ec2.internal\" not found" Jul 22 09:17:51.135714 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:51.135690 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-138.ec2.internal\" not found" Jul 22 09:17:51.236260 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:51.236222 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-138.ec2.internal\" not found" Jul 22 09:17:51.337116 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:51.337059 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-138.ec2.internal\" not found" Jul 22 09:17:51.378667 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.378641 2549 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 09:17:51.405754 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.405733 2549 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-138.ec2.internal" Jul 22 09:17:51.417502 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.417481 2549 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 09:17:51.418253 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.418239 2549 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" Jul 22 09:17:51.429136 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.429117 2549 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 09:17:51.795581 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.795498 2549 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 09:17:51.800627 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.800602 2549 apiserver.go:52] "Watching apiserver" Jul 22 09:17:51.809667 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.809647 2549 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 09:17:51.810545 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.810526 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal","openshift-network-diagnostics/network-check-target-mrw5j","openshift-cluster-node-tuning-operator/tuned-f2mdv","openshift-image-registry/node-ca-rn8j7","openshift-multus/multus-additional-cni-plugins-b5f2z","openshift-multus/multus-jrpbc","openshift-multus/network-metrics-daemon-6sjqx","openshift-network-operator/iptables-alerter-z9599","openshift-ovn-kubernetes/ovnkube-node-bcsgw","kube-system/global-pull-secret-syncer-49l4x","kube-system/konnectivity-agent-7lqtp","kube-system/kube-apiserver-proxy-ip-10-0-138-138.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg","openshift-dns/node-resolver-659hp"] Jul 22 09:17:51.813072 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.813058 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:17:51.813145 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:51.813119 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6sjqx" podUID="54cff721-5d48-4e7a-9771-1b859bc68609" Jul 22 09:17:51.815129 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.815112 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:51.817247 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.817229 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:51.817567 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.817550 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 09:17:51.817648 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.817592 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 09:17:51.817648 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.817605 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9rvmm\"" Jul 22 09:17:51.817744 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.817710 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 09:17:51.819352 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.819336 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f8nvw\"" Jul 22 09:17:51.819440 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.819401 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rn8j7" Jul 22 09:17:51.819440 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.819423 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 09:17:51.819645 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.819627 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 09:17:51.821528 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.821513 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 09:17:51.821582 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.821561 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4sh8g\"" Jul 22 09:17:51.821625 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.821601 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 09:17:51.821672 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.821649 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:51.821980 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.821966 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 09:17:51.823778 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.823764 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jrpbc" Jul 22 09:17:51.825203 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.825185 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b9jtv\"" Jul 22 09:17:51.825420 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.825405 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 09:17:51.825749 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.825732 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 09:17:51.825866 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.825850 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 09:17:51.825935 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.825864 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 09:17:51.826101 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.826088 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:51.826393 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.826217 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sh5rp\"" Jul 22 09:17:51.827435 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.827420 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 09:17:51.828169 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.828155 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 09:17:51.828390 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.828375 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-659hp" Jul 22 09:17:51.828581 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.828565 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 09:17:51.828665 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.828649 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5mxcj\"" Jul 22 09:17:51.828754 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.828739 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 09:17:51.828911 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.828899 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 09:17:51.829431 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.829420 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 09:17:51.829474 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.829429 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 09:17:51.829474 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.829461 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 09:17:51.831672 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.830931 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:17:51.831672 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:51.831018 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrw5j" podUID="2e590737-a6b2-482a-9cf2-23d1add97650" Jul 22 09:17:51.832238 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.832220 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 09:17:51.832490 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.832473 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x867g\"" Jul 22 09:17:51.832610 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.832592 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 09:17:51.833817 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.833797 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:17:51.833899 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:51.833857 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49l4x" podUID="a5680bfb-a669-4640-876e-84a6322b2454" Jul 22 09:17:51.836117 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.836103 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7lqtp" Jul 22 09:17:51.838285 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.838268 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 09:17:51.838422 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.838404 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z9599" Jul 22 09:17:51.838584 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.838521 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 09:17:51.838584 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.838540 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-j8kjf\"" Jul 22 09:17:51.840694 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.840675 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 09:17:51.841720 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.841706 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 09:17:51.841960 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.841942 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 09:17:51.842186 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.842173 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xfkrf\"" Jul 22 09:17:51.907008 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.906991 2549 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 09:17:51.915727 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.915700 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-os-release\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:51.915831 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.915730 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-cni-binary-copy\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:51.915831 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.915757 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-host-run-multus-certs\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:51.915831 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.915780 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dc8d8789-74a0-44ec-8902-44a9650c41ac-ovnkube-config\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:51.915831 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.915802 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dc8d8789-74a0-44ec-8902-44a9650c41ac-ovnkube-script-lib\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:51.915978 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.915840 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zvhx\" (UniqueName: \"kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx\") pod \"network-check-target-mrw5j\" (UID: \"2e590737-a6b2-482a-9cf2-23d1add97650\") " pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:17:51.915978 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.915883 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-run\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:51.915978 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.915905 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a5680bfb-a669-4640-876e-84a6322b2454-kubelet-config\") pod \"global-pull-secret-syncer-49l4x\" (UID: \"a5680bfb-a669-4640-876e-84a6322b2454\") " pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:17:51.915978 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.915921 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:51.915978 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.915935 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-host-var-lib-cni-multus\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:51.915978 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.915952 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-run-systemd\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:51.915978 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.915968 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-mxzpg\" (UID: \"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:51.916235 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.915983 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-system-cni-dir\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:51.916235 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.915996 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-multus-conf-dir\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:51.916235 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916009 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-host-cni-bin\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:51.916235 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916023 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-host-cni-netd\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:51.916235 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916036 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a87cdd8f-1617-4f31-85ab-6ba3294ef8be-hosts-file\") pod \"node-resolver-659hp\" (UID: \"a87cdd8f-1617-4f31-85ab-6ba3294ef8be\") " pod="openshift-dns/node-resolver-659hp" Jul 22 09:17:51.916235 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916066 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5jxj\" (UniqueName: \"kubernetes.io/projected/beb97b8f-7ed2-41fc-87b5-7a0488638e16-kube-api-access-q5jxj\") pod \"node-ca-rn8j7\" (UID: \"beb97b8f-7ed2-41fc-87b5-7a0488638e16\") " pod="openshift-image-registry/node-ca-rn8j7" Jul 22 09:17:51.916235 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916081 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af47e4da-92fc-4f12-abcf-c4c2975d28ba-host-slash\") pod \"iptables-alerter-z9599\" (UID: \"af47e4da-92fc-4f12-abcf-c4c2975d28ba\") " pod="openshift-network-operator/iptables-alerter-z9599" Jul 22 09:17:51.916235 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916104 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-host-kubelet\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:51.916235 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916121 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-host-slash\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:51.916235 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916139 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dc8d8789-74a0-44ec-8902-44a9650c41ac-env-overrides\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:51.916235 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916155 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-host-var-lib-cni-bin\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:51.916235 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916185 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t5dx\" (UniqueName: \"kubernetes.io/projected/fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f-kube-api-access-6t5dx\") pod \"aws-ebs-csi-driver-node-mxzpg\" (UID: \"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:51.916235 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916207 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af47e4da-92fc-4f12-abcf-c4c2975d28ba-iptables-alerter-script\") pod \"iptables-alerter-z9599\" (UID: \"af47e4da-92fc-4f12-abcf-c4c2975d28ba\") " pod="openshift-network-operator/iptables-alerter-z9599" Jul 22 09:17:51.916235 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916226 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-var-lib-openvswitch\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:51.916824 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916255 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djbb2\" (UniqueName: \"kubernetes.io/projected/dc8d8789-74a0-44ec-8902-44a9650c41ac-kube-api-access-djbb2\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:51.916824 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916284 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-lib-modules\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:51.916824 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916321 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7dd0f078-e04e-41fb-925d-09f6e234e4b5-tmp\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:51.916824 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916346 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs\") pod \"network-metrics-daemon-6sjqx\" (UID: \"54cff721-5d48-4e7a-9771-1b859bc68609\") " pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:17:51.916824 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916391 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-system-cni-dir\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:51.916824 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916408 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-etc-kubernetes\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:51.916824 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916437 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-node-log\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:51.916824 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916469 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc8d8789-74a0-44ec-8902-44a9650c41ac-ovn-node-metrics-cert\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:51.916824 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916501 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mxzpg\" (UID: \"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:51.916824 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916525 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f-device-dir\") pod \"aws-ebs-csi-driver-node-mxzpg\" (UID: \"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:51.916824 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916548 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a5680bfb-a669-4640-876e-84a6322b2454-dbus\") pod \"global-pull-secret-syncer-49l4x\" (UID: \"a5680bfb-a669-4640-876e-84a6322b2454\") " pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:17:51.916824 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916575 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-cnibin\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:51.916824 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916597 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-host-run-k8s-cni-cncf-io\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:51.916824 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916622 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:51.917353 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916667 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a87cdd8f-1617-4f31-85ab-6ba3294ef8be-tmp-dir\") pod \"node-resolver-659hp\" (UID: \"a87cdd8f-1617-4f31-85ab-6ba3294ef8be\") " pod="openshift-dns/node-resolver-659hp" Jul 22 09:17:51.917353 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916692 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f-registration-dir\") pod \"aws-ebs-csi-driver-node-mxzpg\" (UID: \"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:51.917353 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916717 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f-sys-fs\") pod \"aws-ebs-csi-driver-node-mxzpg\" (UID: \"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:51.917353 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916739 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-etc-sysctl-conf\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:51.917353 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916762 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/beb97b8f-7ed2-41fc-87b5-7a0488638e16-serviceca\") pod \"node-ca-rn8j7\" (UID: \"beb97b8f-7ed2-41fc-87b5-7a0488638e16\") " pod="openshift-image-registry/node-ca-rn8j7" Jul 22 09:17:51.917353 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916784 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvz68\" (UniqueName: \"kubernetes.io/projected/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-kube-api-access-mvz68\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:51.917353 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916815 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh76s\" (UniqueName: \"kubernetes.io/projected/a87cdd8f-1617-4f31-85ab-6ba3294ef8be-kube-api-access-lh76s\") pod \"node-resolver-659hp\" (UID: \"a87cdd8f-1617-4f31-85ab-6ba3294ef8be\") " pod="openshift-dns/node-resolver-659hp" Jul 22 09:17:51.917353 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916839 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-cni-binary-copy\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:51.917353 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916864 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-host-run-netns\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:51.917353 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916886 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-etc-openvswitch\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:51.917353 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916912 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-etc-sysctl-d\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:51.917353 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916932 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-var-lib-kubelet\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:51.917353 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916947 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-host\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:51.917353 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916969 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret\") pod \"global-pull-secret-syncer-49l4x\" (UID: \"a5680bfb-a669-4640-876e-84a6322b2454\") " pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:17:51.917861 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.916997 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6rr8\" (UniqueName: \"kubernetes.io/projected/af47e4da-92fc-4f12-abcf-c4c2975d28ba-kube-api-access-j6rr8\") pod \"iptables-alerter-z9599\" (UID: \"af47e4da-92fc-4f12-abcf-c4c2975d28ba\") " pod="openshift-network-operator/iptables-alerter-z9599" Jul 22 09:17:51.917861 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917024 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-etc-systemd\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:51.917861 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917045 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/beb97b8f-7ed2-41fc-87b5-7a0488638e16-host\") pod \"node-ca-rn8j7\" (UID: \"beb97b8f-7ed2-41fc-87b5-7a0488638e16\") " pod="openshift-image-registry/node-ca-rn8j7" Jul 22 09:17:51.917861 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917077 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:51.917861 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917103 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-multus-cni-dir\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:51.917861 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917132 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-host-run-netns\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:51.917861 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917155 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-log-socket\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:51.917861 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917169 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f757fe69-cbfe-4254-85cf-07a10840f85b-konnectivity-ca\") pod \"konnectivity-agent-7lqtp\" (UID: \"f757fe69-cbfe-4254-85cf-07a10840f85b\") " pod="kube-system/konnectivity-agent-7lqtp" Jul 22 09:17:51.917861 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917182 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f-socket-dir\") pod \"aws-ebs-csi-driver-node-mxzpg\" (UID: \"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:51.917861 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917196 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-etc-kubernetes\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:51.917861 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917208 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-sys\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:51.917861 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917225 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7dd0f078-e04e-41fb-925d-09f6e234e4b5-etc-tuned\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:51.917861 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917238 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v4qb\" (UniqueName: \"kubernetes.io/projected/7dd0f078-e04e-41fb-925d-09f6e234e4b5-kube-api-access-6v4qb\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:51.917861 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917251 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-cnibin\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:51.918238 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917265 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-etc-modprobe-d\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:51.918238 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917279 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:51.918238 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917323 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwdm9\" (UniqueName: \"kubernetes.io/projected/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-kube-api-access-nwdm9\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:51.918238 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917353 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-multus-socket-dir-parent\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:51.918238 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917373 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-hostroot\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:51.918238 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917398 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-multus-daemon-config\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:51.918238 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917421 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-systemd-units\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:51.918238 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917448 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-run-ovn\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:51.918238 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917471 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f757fe69-cbfe-4254-85cf-07a10840f85b-agent-certs\") pod \"konnectivity-agent-7lqtp\" (UID: \"f757fe69-cbfe-4254-85cf-07a10840f85b\") " pod="kube-system/konnectivity-agent-7lqtp" Jul 22 09:17:51.918238 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917493 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-os-release\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:51.918238 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917514 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-host-var-lib-kubelet\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:51.918238 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917545 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-run-openvswitch\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:51.918238 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917569 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:51.918238 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917594 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-etc-sysconfig\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:51.918693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:51.917624 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bjz6\" (UniqueName: \"kubernetes.io/projected/54cff721-5d48-4e7a-9771-1b859bc68609-kube-api-access-7bjz6\") pod \"network-metrics-daemon-6sjqx\" (UID: \"54cff721-5d48-4e7a-9771-1b859bc68609\") " pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:17:52.017924 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.017886 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-mxzpg\" (UID: \"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:52.017924 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.017920 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-system-cni-dir\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.018175 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.017957 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-multus-conf-dir\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.018175 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.017986 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-host-cni-bin\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.018175 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018001 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-host-cni-netd\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.018175 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018015 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a87cdd8f-1617-4f31-85ab-6ba3294ef8be-hosts-file\") pod \"node-resolver-659hp\" (UID: \"a87cdd8f-1617-4f31-85ab-6ba3294ef8be\") " pod="openshift-dns/node-resolver-659hp" Jul 22 09:17:52.018175 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018036 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5jxj\" (UniqueName: \"kubernetes.io/projected/beb97b8f-7ed2-41fc-87b5-7a0488638e16-kube-api-access-q5jxj\") pod \"node-ca-rn8j7\" (UID: \"beb97b8f-7ed2-41fc-87b5-7a0488638e16\") " pod="openshift-image-registry/node-ca-rn8j7" Jul 22 09:17:52.018175 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018045 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-multus-conf-dir\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.018175 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018045 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-mxzpg\" (UID: \"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:52.018175 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018059 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af47e4da-92fc-4f12-abcf-c4c2975d28ba-host-slash\") pod \"iptables-alerter-z9599\" (UID: \"af47e4da-92fc-4f12-abcf-c4c2975d28ba\") " pod="openshift-network-operator/iptables-alerter-z9599" Jul 22 09:17:52.018175 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018065 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-host-cni-bin\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.018175 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018086 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-system-cni-dir\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.018175 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018089 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-host-cni-netd\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.018175 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018092 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-host-kubelet\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.018175 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018099 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af47e4da-92fc-4f12-abcf-c4c2975d28ba-host-slash\") pod \"iptables-alerter-z9599\" (UID: \"af47e4da-92fc-4f12-abcf-c4c2975d28ba\") " pod="openshift-network-operator/iptables-alerter-z9599" Jul 22 09:17:52.018175 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018121 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-host-kubelet\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.018175 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018112 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a87cdd8f-1617-4f31-85ab-6ba3294ef8be-hosts-file\") pod \"node-resolver-659hp\" (UID: \"a87cdd8f-1617-4f31-85ab-6ba3294ef8be\") " pod="openshift-dns/node-resolver-659hp" Jul 22 09:17:52.018877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018132 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-host-slash\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.018877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018156 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dc8d8789-74a0-44ec-8902-44a9650c41ac-env-overrides\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.018877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018156 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-host-slash\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.018877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018245 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-host-var-lib-cni-bin\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.018877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018266 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t5dx\" (UniqueName: \"kubernetes.io/projected/fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f-kube-api-access-6t5dx\") pod \"aws-ebs-csi-driver-node-mxzpg\" (UID: \"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:52.018877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018280 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af47e4da-92fc-4f12-abcf-c4c2975d28ba-iptables-alerter-script\") pod \"iptables-alerter-z9599\" (UID: \"af47e4da-92fc-4f12-abcf-c4c2975d28ba\") " pod="openshift-network-operator/iptables-alerter-z9599" Jul 22 09:17:52.018877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018295 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-var-lib-openvswitch\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.018877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018326 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djbb2\" (UniqueName: \"kubernetes.io/projected/dc8d8789-74a0-44ec-8902-44a9650c41ac-kube-api-access-djbb2\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.018877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018376 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-host-var-lib-cni-bin\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.018877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018406 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-lib-modules\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.018877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018409 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-var-lib-openvswitch\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.018877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018430 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7dd0f078-e04e-41fb-925d-09f6e234e4b5-tmp\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.018877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018454 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs\") pod \"network-metrics-daemon-6sjqx\" (UID: \"54cff721-5d48-4e7a-9771-1b859bc68609\") " pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:17:52.018877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018479 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-system-cni-dir\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:52.019479 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018502 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-etc-kubernetes\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.019479 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018525 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-node-log\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.019479 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018540 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-lib-modules\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.019479 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:52.018554 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:17:52.019479 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018584 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-node-log\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.019479 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018582 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-etc-kubernetes\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.019479 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018548 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc8d8789-74a0-44ec-8902-44a9650c41ac-ovn-node-metrics-cert\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.019479 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018619 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-system-cni-dir\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:52.019479 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018623 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mxzpg\" (UID: \"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:52.019479 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:52.018644 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs podName:54cff721-5d48-4e7a-9771-1b859bc68609 nodeName:}" failed. No retries permitted until 2026-07-22 09:17:52.518623602 +0000 UTC m=+2.982418045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs") pod "network-metrics-daemon-6sjqx" (UID: "54cff721-5d48-4e7a-9771-1b859bc68609") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:17:52.019479 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018652 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dc8d8789-74a0-44ec-8902-44a9650c41ac-env-overrides\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.019479 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018669 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mxzpg\" (UID: \"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:52.019479 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018682 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f-device-dir\") pod \"aws-ebs-csi-driver-node-mxzpg\" (UID: \"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:52.019479 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018683 2549 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 09:17:52.019479 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018724 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a5680bfb-a669-4640-876e-84a6322b2454-dbus\") pod \"global-pull-secret-syncer-49l4x\" (UID: \"a5680bfb-a669-4640-876e-84a6322b2454\") " pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:17:52.020152 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018748 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-cnibin\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.020152 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018754 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f-device-dir\") pod \"aws-ebs-csi-driver-node-mxzpg\" (UID: \"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:52.020152 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018772 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-host-run-k8s-cni-cncf-io\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.020152 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018796 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.020152 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018803 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-cnibin\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.020152 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018819 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a87cdd8f-1617-4f31-85ab-6ba3294ef8be-tmp-dir\") pod \"node-resolver-659hp\" (UID: \"a87cdd8f-1617-4f31-85ab-6ba3294ef8be\") " pod="openshift-dns/node-resolver-659hp" Jul 22 09:17:52.020152 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018826 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-host-run-k8s-cni-cncf-io\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.020152 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018837 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.020152 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018845 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f-registration-dir\") pod \"aws-ebs-csi-driver-node-mxzpg\" (UID: \"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:52.020152 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018860 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a5680bfb-a669-4640-876e-84a6322b2454-dbus\") pod \"global-pull-secret-syncer-49l4x\" (UID: \"a5680bfb-a669-4640-876e-84a6322b2454\") " pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:17:52.020152 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018872 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f-sys-fs\") pod \"aws-ebs-csi-driver-node-mxzpg\" (UID: \"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:52.020152 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018877 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af47e4da-92fc-4f12-abcf-c4c2975d28ba-iptables-alerter-script\") pod \"iptables-alerter-z9599\" (UID: \"af47e4da-92fc-4f12-abcf-c4c2975d28ba\") " pod="openshift-network-operator/iptables-alerter-z9599" Jul 22 09:17:52.020152 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018889 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f-registration-dir\") pod \"aws-ebs-csi-driver-node-mxzpg\" (UID: \"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:52.020152 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018897 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-etc-sysctl-conf\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.020152 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018920 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/beb97b8f-7ed2-41fc-87b5-7a0488638e16-serviceca\") pod \"node-ca-rn8j7\" (UID: \"beb97b8f-7ed2-41fc-87b5-7a0488638e16\") " pod="openshift-image-registry/node-ca-rn8j7" Jul 22 09:17:52.020877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018915 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f-sys-fs\") pod \"aws-ebs-csi-driver-node-mxzpg\" (UID: \"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:52.020877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018942 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvz68\" (UniqueName: \"kubernetes.io/projected/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-kube-api-access-mvz68\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.020877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.018982 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh76s\" (UniqueName: \"kubernetes.io/projected/a87cdd8f-1617-4f31-85ab-6ba3294ef8be-kube-api-access-lh76s\") pod \"node-resolver-659hp\" (UID: \"a87cdd8f-1617-4f31-85ab-6ba3294ef8be\") " pod="openshift-dns/node-resolver-659hp" Jul 22 09:17:52.020877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019005 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-cni-binary-copy\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:52.020877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019021 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-etc-sysctl-conf\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.020877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019028 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-host-run-netns\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.020877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019052 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-etc-openvswitch\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.020877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019073 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-etc-sysctl-d\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.020877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019097 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-var-lib-kubelet\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.020877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019108 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a87cdd8f-1617-4f31-85ab-6ba3294ef8be-tmp-dir\") pod \"node-resolver-659hp\" (UID: \"a87cdd8f-1617-4f31-85ab-6ba3294ef8be\") " pod="openshift-dns/node-resolver-659hp" Jul 22 09:17:52.020877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019120 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-host\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.020877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019144 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret\") pod \"global-pull-secret-syncer-49l4x\" (UID: \"a5680bfb-a669-4640-876e-84a6322b2454\") " pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:17:52.020877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019168 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6rr8\" (UniqueName: \"kubernetes.io/projected/af47e4da-92fc-4f12-abcf-c4c2975d28ba-kube-api-access-j6rr8\") pod \"iptables-alerter-z9599\" (UID: \"af47e4da-92fc-4f12-abcf-c4c2975d28ba\") " pod="openshift-network-operator/iptables-alerter-z9599" Jul 22 09:17:52.020877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019173 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-etc-openvswitch\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.021516 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019186 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-host-run-netns\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.021516 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019219 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-host\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.021516 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019255 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-etc-systemd\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.021516 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:52.019269 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 09:17:52.021516 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019282 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/beb97b8f-7ed2-41fc-87b5-7a0488638e16-host\") pod \"node-ca-rn8j7\" (UID: \"beb97b8f-7ed2-41fc-87b5-7a0488638e16\") " pod="openshift-image-registry/node-ca-rn8j7" Jul 22 09:17:52.021516 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019298 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-etc-sysctl-d\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.021516 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019324 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:52.021516 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:52.019333 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret podName:a5680bfb-a669-4640-876e-84a6322b2454 nodeName:}" failed. No retries permitted until 2026-07-22 09:17:52.519317294 +0000 UTC m=+2.983111725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret") pod "global-pull-secret-syncer-49l4x" (UID: "a5680bfb-a669-4640-876e-84a6322b2454") : object "kube-system"/"original-pull-secret" not registered Jul 22 09:17:52.021516 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019365 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/beb97b8f-7ed2-41fc-87b5-7a0488638e16-host\") pod \"node-ca-rn8j7\" (UID: \"beb97b8f-7ed2-41fc-87b5-7a0488638e16\") " pod="openshift-image-registry/node-ca-rn8j7" Jul 22 09:17:52.021516 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019369 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-etc-systemd\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.021516 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019372 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-multus-cni-dir\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.021516 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019404 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-host-run-netns\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.021516 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019427 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-log-socket\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.021516 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019450 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-multus-cni-dir\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.021516 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019458 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-host-run-netns\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.022188 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019454 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f757fe69-cbfe-4254-85cf-07a10840f85b-konnectivity-ca\") pod \"konnectivity-agent-7lqtp\" (UID: \"f757fe69-cbfe-4254-85cf-07a10840f85b\") " pod="kube-system/konnectivity-agent-7lqtp" Jul 22 09:17:52.022188 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019457 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/beb97b8f-7ed2-41fc-87b5-7a0488638e16-serviceca\") pod \"node-ca-rn8j7\" (UID: \"beb97b8f-7ed2-41fc-87b5-7a0488638e16\") " pod="openshift-image-registry/node-ca-rn8j7" Jul 22 09:17:52.022188 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019497 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f-socket-dir\") pod \"aws-ebs-csi-driver-node-mxzpg\" (UID: \"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:52.022188 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019429 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-var-lib-kubelet\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.022188 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019499 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-log-socket\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.022188 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019532 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-etc-kubernetes\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.022188 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019569 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-sys\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.022188 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019588 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f-socket-dir\") pod \"aws-ebs-csi-driver-node-mxzpg\" (UID: \"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:52.022188 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019591 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-cni-binary-copy\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:52.022188 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019593 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7dd0f078-e04e-41fb-925d-09f6e234e4b5-etc-tuned\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.022188 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019622 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-sys\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.022188 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019638 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v4qb\" (UniqueName: \"kubernetes.io/projected/7dd0f078-e04e-41fb-925d-09f6e234e4b5-kube-api-access-6v4qb\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.022188 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019632 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-etc-kubernetes\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.022188 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019672 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-cnibin\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:52.022188 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019696 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-etc-modprobe-d\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.022816 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019720 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:52.022816 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019744 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwdm9\" (UniqueName: \"kubernetes.io/projected/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-kube-api-access-nwdm9\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:52.022816 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019770 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-multus-socket-dir-parent\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.022816 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019794 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-hostroot\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.022816 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019817 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-multus-daemon-config\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.022816 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019823 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:52.022816 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019840 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-systemd-units\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.022816 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019862 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-run-ovn\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.022816 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019883 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-hostroot\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.022816 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019885 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f757fe69-cbfe-4254-85cf-07a10840f85b-agent-certs\") pod \"konnectivity-agent-7lqtp\" (UID: \"f757fe69-cbfe-4254-85cf-07a10840f85b\") " pod="kube-system/konnectivity-agent-7lqtp" Jul 22 09:17:52.022816 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019927 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-os-release\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:52.022816 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019947 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:52.022816 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019953 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f757fe69-cbfe-4254-85cf-07a10840f85b-konnectivity-ca\") pod \"konnectivity-agent-7lqtp\" (UID: \"f757fe69-cbfe-4254-85cf-07a10840f85b\") " pod="kube-system/konnectivity-agent-7lqtp" Jul 22 09:17:52.022816 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019952 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-host-var-lib-kubelet\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.023203 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.019997 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-host-var-lib-kubelet\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.023203 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020006 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-run-openvswitch\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.023203 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020010 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-multus-socket-dir-parent\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.023203 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020041 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.023203 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020055 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-cnibin\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:52.023203 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020056 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-os-release\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:52.023203 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020068 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-etc-sysconfig\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.023203 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020092 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-systemd-units\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.023203 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020103 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bjz6\" (UniqueName: \"kubernetes.io/projected/54cff721-5d48-4e7a-9771-1b859bc68609-kube-api-access-7bjz6\") pod \"network-metrics-daemon-6sjqx\" (UID: \"54cff721-5d48-4e7a-9771-1b859bc68609\") " pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:17:52.023203 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020109 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-etc-sysconfig\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.023203 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020149 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-run-ovn\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.023203 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020153 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-os-release\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.023203 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020156 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.023203 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020185 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-run-openvswitch\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.023608 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020222 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-cni-binary-copy\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.023608 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020247 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-host-run-multus-certs\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.023608 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020271 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dc8d8789-74a0-44ec-8902-44a9650c41ac-ovnkube-config\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.023608 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020297 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dc8d8789-74a0-44ec-8902-44a9650c41ac-ovnkube-script-lib\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.023608 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020381 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvhx\" (UniqueName: \"kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx\") pod \"network-check-target-mrw5j\" (UID: \"2e590737-a6b2-482a-9cf2-23d1add97650\") " pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:17:52.023608 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020413 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-run\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.023608 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020438 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a5680bfb-a669-4640-876e-84a6322b2454-kubelet-config\") pod \"global-pull-secret-syncer-49l4x\" (UID: \"a5680bfb-a669-4640-876e-84a6322b2454\") " pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:17:52.023608 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020500 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-run\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.023608 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020529 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:52.023608 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020563 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-host-var-lib-cni-multus\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.023608 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020588 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-run-systemd\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.023608 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020658 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dc8d8789-74a0-44ec-8902-44a9650c41ac-run-systemd\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.023608 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020702 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a5680bfb-a669-4640-876e-84a6322b2454-kubelet-config\") pod \"global-pull-secret-syncer-49l4x\" (UID: \"a5680bfb-a669-4640-876e-84a6322b2454\") " pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:17:52.023608 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020721 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-cni-binary-copy\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.023608 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020756 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-host-run-multus-certs\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.024013 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020823 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dc8d8789-74a0-44ec-8902-44a9650c41ac-ovnkube-config\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.024013 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020892 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-os-release\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.024013 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020963 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-multus-daemon-config\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.024013 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.020968 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-host-var-lib-cni-multus\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.024013 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.021158 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dc8d8789-74a0-44ec-8902-44a9650c41ac-ovnkube-script-lib\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.024013 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.021327 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:52.024013 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.021393 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7dd0f078-e04e-41fb-925d-09f6e234e4b5-etc-modprobe-d\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.024013 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.022253 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7dd0f078-e04e-41fb-925d-09f6e234e4b5-etc-tuned\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.024013 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.022463 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc8d8789-74a0-44ec-8902-44a9650c41ac-ovn-node-metrics-cert\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.024013 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.022656 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f757fe69-cbfe-4254-85cf-07a10840f85b-agent-certs\") pod \"konnectivity-agent-7lqtp\" (UID: \"f757fe69-cbfe-4254-85cf-07a10840f85b\") " pod="kube-system/konnectivity-agent-7lqtp" Jul 22 09:17:52.024013 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.022740 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7dd0f078-e04e-41fb-925d-09f6e234e4b5-tmp\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.026699 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.026675 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5jxj\" (UniqueName: \"kubernetes.io/projected/beb97b8f-7ed2-41fc-87b5-7a0488638e16-kube-api-access-q5jxj\") pod \"node-ca-rn8j7\" (UID: \"beb97b8f-7ed2-41fc-87b5-7a0488638e16\") " pod="openshift-image-registry/node-ca-rn8j7" Jul 22 09:17:52.027921 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.027898 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t5dx\" (UniqueName: \"kubernetes.io/projected/fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f-kube-api-access-6t5dx\") pod \"aws-ebs-csi-driver-node-mxzpg\" (UID: \"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:52.038904 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:52.038882 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 09:17:52.039011 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:52.038912 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 09:17:52.039011 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:52.038928 2549 projected.go:196] Error preparing data for projected volume kube-api-access-5zvhx for pod openshift-network-diagnostics/network-check-target-mrw5j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:17:52.039011 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:52.039010 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx podName:2e590737-a6b2-482a-9cf2-23d1add97650 nodeName:}" failed. No retries permitted until 2026-07-22 09:17:52.538988642 +0000 UTC m=+3.002783073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5zvhx" (UniqueName: "kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx") pod "network-check-target-mrw5j" (UID: "2e590737-a6b2-482a-9cf2-23d1add97650") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:17:52.040830 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.040805 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6rr8\" (UniqueName: \"kubernetes.io/projected/af47e4da-92fc-4f12-abcf-c4c2975d28ba-kube-api-access-j6rr8\") pod \"iptables-alerter-z9599\" (UID: \"af47e4da-92fc-4f12-abcf-c4c2975d28ba\") " pod="openshift-network-operator/iptables-alerter-z9599" Jul 22 09:17:52.041211 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.041193 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v4qb\" (UniqueName: \"kubernetes.io/projected/7dd0f078-e04e-41fb-925d-09f6e234e4b5-kube-api-access-6v4qb\") pod \"tuned-f2mdv\" (UID: \"7dd0f078-e04e-41fb-925d-09f6e234e4b5\") " pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.041294 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.041275 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh76s\" (UniqueName: \"kubernetes.io/projected/a87cdd8f-1617-4f31-85ab-6ba3294ef8be-kube-api-access-lh76s\") pod \"node-resolver-659hp\" (UID: \"a87cdd8f-1617-4f31-85ab-6ba3294ef8be\") " pod="openshift-dns/node-resolver-659hp" Jul 22 09:17:52.041563 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.041542 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bjz6\" (UniqueName: \"kubernetes.io/projected/54cff721-5d48-4e7a-9771-1b859bc68609-kube-api-access-7bjz6\") pod \"network-metrics-daemon-6sjqx\" (UID: \"54cff721-5d48-4e7a-9771-1b859bc68609\") " pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:17:52.041628 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.041610 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djbb2\" (UniqueName: \"kubernetes.io/projected/dc8d8789-74a0-44ec-8902-44a9650c41ac-kube-api-access-djbb2\") pod \"ovnkube-node-bcsgw\" (UID: \"dc8d8789-74a0-44ec-8902-44a9650c41ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.041885 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.041871 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvz68\" (UniqueName: \"kubernetes.io/projected/9ed89b08-ed62-4f3b-96b1-3bdba846b4f3-kube-api-access-mvz68\") pod \"multus-jrpbc\" (UID: \"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3\") " pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.042459 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.042445 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwdm9\" (UniqueName: \"kubernetes.io/projected/f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6-kube-api-access-nwdm9\") pod \"multus-additional-cni-plugins-b5f2z\" (UID: \"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6\") " pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:52.139489 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.139406 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" Jul 22 09:17:52.157710 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.157688 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" Jul 22 09:17:52.172201 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.172185 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rn8j7" Jul 22 09:17:52.183796 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.183777 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b5f2z" Jul 22 09:17:52.190866 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.190830 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jrpbc" Jul 22 09:17:52.204430 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.204413 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:17:52.206489 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.206471 2549 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 09:17:52.216107 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.216090 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-659hp" Jul 22 09:17:52.221775 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.221758 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7lqtp" Jul 22 09:17:52.226449 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.226432 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z9599" Jul 22 09:17:52.228895 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.228878 2549 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 09:17:52.246815 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.246790 2549 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 09:17:52.266025 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.266006 2549 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-g5ppp" Jul 22 09:17:52.278364 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.278344 2549 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g5ppp" Jul 22 09:17:52.523096 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.523031 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs\") pod \"network-metrics-daemon-6sjqx\" (UID: \"54cff721-5d48-4e7a-9771-1b859bc68609\") " pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:17:52.523096 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.523070 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret\") pod \"global-pull-secret-syncer-49l4x\" (UID: \"a5680bfb-a669-4640-876e-84a6322b2454\") " pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:17:52.523253 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:52.523170 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 09:17:52.523253 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:52.523172 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:17:52.523253 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:52.523233 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs podName:54cff721-5d48-4e7a-9771-1b859bc68609 nodeName:}" failed. No retries permitted until 2026-07-22 09:17:53.523219182 +0000 UTC m=+3.987013613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs") pod "network-metrics-daemon-6sjqx" (UID: "54cff721-5d48-4e7a-9771-1b859bc68609") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:17:52.523253 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:52.523249 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret podName:a5680bfb-a669-4640-876e-84a6322b2454 nodeName:}" failed. No retries permitted until 2026-07-22 09:17:53.523243161 +0000 UTC m=+3.987037576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret") pod "global-pull-secret-syncer-49l4x" (UID: "a5680bfb-a669-4640-876e-84a6322b2454") : object "kube-system"/"original-pull-secret" not registered Jul 22 09:17:52.624047 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.624013 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvhx\" (UniqueName: \"kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx\") pod \"network-check-target-mrw5j\" (UID: \"2e590737-a6b2-482a-9cf2-23d1add97650\") " pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:17:52.624273 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:52.624147 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 09:17:52.624273 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:52.624161 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 09:17:52.624273 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:52.624170 2549 projected.go:196] Error preparing data for projected volume kube-api-access-5zvhx for pod openshift-network-diagnostics/network-check-target-mrw5j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:17:52.624273 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:52.624220 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx podName:2e590737-a6b2-482a-9cf2-23d1add97650 nodeName:}" failed. No retries permitted until 2026-07-22 09:17:53.624207777 +0000 UTC m=+4.088002206 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5zvhx" (UniqueName: "kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx") pod "network-check-target-mrw5j" (UID: "2e590737-a6b2-482a-9cf2-23d1add97650") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:17:52.703866 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:52.703813 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb97b8f_7ed2_41fc_87b5_7a0488638e16.slice/crio-a97e97435d5efdd61be4f3ef68223d364d1e354df4c9513820a1634550700c1d WatchSource:0}: Error finding container a97e97435d5efdd61be4f3ef68223d364d1e354df4c9513820a1634550700c1d: Status 404 returned error can't find the container with id a97e97435d5efdd61be4f3ef68223d364d1e354df4c9513820a1634550700c1d Jul 22 09:17:52.704874 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:52.704850 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8d8789_74a0_44ec_8902_44a9650c41ac.slice/crio-e91108f22235fc0957502770161dff5173db955c6cea34c63242420278de98d3 WatchSource:0}: Error finding container e91108f22235fc0957502770161dff5173db955c6cea34c63242420278de98d3: Status 404 returned error can't find the container with id e91108f22235fc0957502770161dff5173db955c6cea34c63242420278de98d3 Jul 22 09:17:52.705799 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:52.705770 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd9efb5_2c5e_4810_936b_d4c29fe3cd1f.slice/crio-9ab27563dfc83454e682a12de56c61672d2e4f67301ad8e689c07d5550dad8a9 WatchSource:0}: Error finding container 9ab27563dfc83454e682a12de56c61672d2e4f67301ad8e689c07d5550dad8a9: Status 404 returned error can't find the container with id 9ab27563dfc83454e682a12de56c61672d2e4f67301ad8e689c07d5550dad8a9 Jul 22 09:17:52.706216 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:52.706110 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69d8767_1133_4d2d_9ebd_5fe2a5fb6ec6.slice/crio-5385d1f909e1da3969b860a36382b642905e330fd9c49dd78c611d2a1f9f99cd WatchSource:0}: Error finding container 5385d1f909e1da3969b860a36382b642905e330fd9c49dd78c611d2a1f9f99cd: Status 404 returned error can't find the container with id 5385d1f909e1da3969b860a36382b642905e330fd9c49dd78c611d2a1f9f99cd Jul 22 09:17:52.713575 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:52.713554 2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 09:17:52.735072 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:52.735049 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87cdd8f_1617_4f31_85ab_6ba3294ef8be.slice/crio-e55021abb64808c72a0020f8ce7a628449ce7cfc555f80987a95f4b3592c5ed8 WatchSource:0}: Error finding container e55021abb64808c72a0020f8ce7a628449ce7cfc555f80987a95f4b3592c5ed8: Status 404 returned error can't find the container with id e55021abb64808c72a0020f8ce7a628449ce7cfc555f80987a95f4b3592c5ed8 Jul 22 09:17:52.740809 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:52.740790 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed89b08_ed62_4f3b_96b1_3bdba846b4f3.slice/crio-17cf45bcdde8527901a0520e6cea7b52f2a9e0dde93097ccec9a372844520786 WatchSource:0}: Error finding container 17cf45bcdde8527901a0520e6cea7b52f2a9e0dde93097ccec9a372844520786: Status 404 returned error can't find the container with id 17cf45bcdde8527901a0520e6cea7b52f2a9e0dde93097ccec9a372844520786 Jul 22 09:17:52.776762 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:52.776732 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf757fe69_cbfe_4254_85cf_07a10840f85b.slice/crio-e6e05bd9fc6afd3850542e420a0796fe18daadc5f63b9bccb8f15606e4af31fd WatchSource:0}: Error finding container e6e05bd9fc6afd3850542e420a0796fe18daadc5f63b9bccb8f15606e4af31fd: Status 404 returned error can't find the container with id e6e05bd9fc6afd3850542e420a0796fe18daadc5f63b9bccb8f15606e4af31fd Jul 22 09:17:52.799975 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:52.799947 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf47e4da_92fc_4f12_abcf_c4c2975d28ba.slice/crio-d627d695227e478bc3e9b37a3873428d1d0a6d84df1ec33115a28e9397a0cc12 WatchSource:0}: Error finding container d627d695227e478bc3e9b37a3873428d1d0a6d84df1ec33115a28e9397a0cc12: Status 404 returned error can't find the container with id d627d695227e478bc3e9b37a3873428d1d0a6d84df1ec33115a28e9397a0cc12 Jul 22 09:17:52.821072 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:52.821051 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd0f078_e04e_41fb_925d_09f6e234e4b5.slice/crio-7ebe3c819398dc50beda368444c7d99fd423858654961fcc564748f0ccd425a4 WatchSource:0}: Error finding container 7ebe3c819398dc50beda368444c7d99fd423858654961fcc564748f0ccd425a4: Status 404 returned error can't find the container with id 7ebe3c819398dc50beda368444c7d99fd423858654961fcc564748f0ccd425a4 Jul 22 09:17:53.019779 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:17:53.019734 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c71e61550bcbc1af7cc7ad68e31b2b.slice/crio-664e2dfc6755118330e8e6d074c5e834f7cf47c5c24a9f638002b8dd1e377013 WatchSource:0}: Error finding container 664e2dfc6755118330e8e6d074c5e834f7cf47c5c24a9f638002b8dd1e377013: Status 404 returned error can't find the container with id 664e2dfc6755118330e8e6d074c5e834f7cf47c5c24a9f638002b8dd1e377013 Jul 22 09:17:53.281422 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:53.281377 2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 09:12:52 +0000 UTC" deadline="2028-03-21 06:43:42.213854543 +0000 UTC" Jul 22 09:17:53.281422 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:53.281428 2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14589h25m48.932431436s" Jul 22 09:17:53.333776 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:53.333745 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:17:53.333951 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:53.333875 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49l4x" podUID="a5680bfb-a669-4640-876e-84a6322b2454" Jul 22 09:17:53.334295 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:53.334276 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:17:53.334401 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:53.333695 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:17:53.334457 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:53.334420 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrw5j" podUID="2e590737-a6b2-482a-9cf2-23d1add97650" Jul 22 09:17:53.334610 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:53.334576 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6sjqx" podUID="54cff721-5d48-4e7a-9771-1b859bc68609" Jul 22 09:17:53.348028 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:53.347933 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" event={"ID":"16c453564e70f602f26f88ff0da5d012","Type":"ContainerStarted","Data":"06e229c6c7091eb791c18f45a5bea23f15cdaecd62feb522c461c87ae9adb324"} Jul 22 09:17:53.352224 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:53.352194 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-138.ec2.internal" event={"ID":"22c71e61550bcbc1af7cc7ad68e31b2b","Type":"ContainerStarted","Data":"664e2dfc6755118330e8e6d074c5e834f7cf47c5c24a9f638002b8dd1e377013"} Jul 22 09:17:53.357010 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:53.356952 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" event={"ID":"7dd0f078-e04e-41fb-925d-09f6e234e4b5","Type":"ContainerStarted","Data":"7ebe3c819398dc50beda368444c7d99fd423858654961fcc564748f0ccd425a4"} Jul 22 09:17:53.359468 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:53.359394 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z9599" event={"ID":"af47e4da-92fc-4f12-abcf-c4c2975d28ba","Type":"ContainerStarted","Data":"d627d695227e478bc3e9b37a3873428d1d0a6d84df1ec33115a28e9397a0cc12"} Jul 22 09:17:53.363382 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:53.363355 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" event={"ID":"dc8d8789-74a0-44ec-8902-44a9650c41ac","Type":"ContainerStarted","Data":"e91108f22235fc0957502770161dff5173db955c6cea34c63242420278de98d3"} Jul 22 09:17:53.370373 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:53.370327 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" event={"ID":"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f","Type":"ContainerStarted","Data":"9ab27563dfc83454e682a12de56c61672d2e4f67301ad8e689c07d5550dad8a9"} Jul 22 09:17:53.372960 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:53.372936 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7lqtp" event={"ID":"f757fe69-cbfe-4254-85cf-07a10840f85b","Type":"ContainerStarted","Data":"e6e05bd9fc6afd3850542e420a0796fe18daadc5f63b9bccb8f15606e4af31fd"} Jul 22 09:17:53.376807 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:53.376783 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jrpbc" event={"ID":"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3","Type":"ContainerStarted","Data":"17cf45bcdde8527901a0520e6cea7b52f2a9e0dde93097ccec9a372844520786"} Jul 22 09:17:53.383973 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:53.383916 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-659hp" event={"ID":"a87cdd8f-1617-4f31-85ab-6ba3294ef8be","Type":"ContainerStarted","Data":"e55021abb64808c72a0020f8ce7a628449ce7cfc555f80987a95f4b3592c5ed8"} Jul 22 09:17:53.387037 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:53.387012 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5f2z" event={"ID":"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6","Type":"ContainerStarted","Data":"5385d1f909e1da3969b860a36382b642905e330fd9c49dd78c611d2a1f9f99cd"} Jul 22 09:17:53.390297 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:53.390275 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rn8j7" event={"ID":"beb97b8f-7ed2-41fc-87b5-7a0488638e16","Type":"ContainerStarted","Data":"a97e97435d5efdd61be4f3ef68223d364d1e354df4c9513820a1634550700c1d"} Jul 22 09:17:53.532560 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:53.532474 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs\") pod \"network-metrics-daemon-6sjqx\" (UID: \"54cff721-5d48-4e7a-9771-1b859bc68609\") " pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:17:53.532560 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:53.532539 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret\") pod \"global-pull-secret-syncer-49l4x\" (UID: \"a5680bfb-a669-4640-876e-84a6322b2454\") " pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:17:53.532778 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:53.532692 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 09:17:53.532778 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:53.532759 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret podName:a5680bfb-a669-4640-876e-84a6322b2454 nodeName:}" failed. No retries permitted until 2026-07-22 09:17:55.532740991 +0000 UTC m=+5.996535428 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret") pod "global-pull-secret-syncer-49l4x" (UID: "a5680bfb-a669-4640-876e-84a6322b2454") : object "kube-system"/"original-pull-secret" not registered Jul 22 09:17:53.533289 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:53.533181 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:17:53.533289 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:53.533238 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs podName:54cff721-5d48-4e7a-9771-1b859bc68609 nodeName:}" failed. No retries permitted until 2026-07-22 09:17:55.533222166 +0000 UTC m=+5.997016596 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs") pod "network-metrics-daemon-6sjqx" (UID: "54cff721-5d48-4e7a-9771-1b859bc68609") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:17:53.633978 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:53.633915 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvhx\" (UniqueName: \"kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx\") pod \"network-check-target-mrw5j\" (UID: \"2e590737-a6b2-482a-9cf2-23d1add97650\") " pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:17:53.634158 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:53.634096 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 09:17:53.634158 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:53.634122 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 09:17:53.634158 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:53.634138 2549 projected.go:196] Error preparing data for projected volume kube-api-access-5zvhx for pod openshift-network-diagnostics/network-check-target-mrw5j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:17:53.634345 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:53.634199 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx podName:2e590737-a6b2-482a-9cf2-23d1add97650 nodeName:}" failed. No retries permitted until 2026-07-22 09:17:55.634179658 +0000 UTC m=+6.097974093 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5zvhx" (UniqueName: "kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx") pod "network-check-target-mrw5j" (UID: "2e590737-a6b2-482a-9cf2-23d1add97650") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:17:54.282515 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:54.282395 2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 09:12:52 +0000 UTC" deadline="2028-02-09 16:18:44.910429813 +0000 UTC" Jul 22 09:17:54.282515 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:54.282438 2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13615h0m50.627996788s" Jul 22 09:17:55.334314 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:55.334245 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:17:55.334778 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:55.334432 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6sjqx" podUID="54cff721-5d48-4e7a-9771-1b859bc68609" Jul 22 09:17:55.334902 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:55.334881 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:17:55.335001 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:55.334982 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49l4x" podUID="a5680bfb-a669-4640-876e-84a6322b2454" Jul 22 09:17:55.335057 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:55.334172 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:17:55.335117 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:55.335079 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrw5j" podUID="2e590737-a6b2-482a-9cf2-23d1add97650" Jul 22 09:17:55.555557 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:55.555514 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret\") pod \"global-pull-secret-syncer-49l4x\" (UID: \"a5680bfb-a669-4640-876e-84a6322b2454\") " pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:17:55.555737 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:55.555613 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs\") pod \"network-metrics-daemon-6sjqx\" (UID: \"54cff721-5d48-4e7a-9771-1b859bc68609\") " pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:17:55.555800 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:55.555749 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:17:55.555855 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:55.555810 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs podName:54cff721-5d48-4e7a-9771-1b859bc68609 nodeName:}" failed. No retries permitted until 2026-07-22 09:17:59.55579166 +0000 UTC m=+10.019586098 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs") pod "network-metrics-daemon-6sjqx" (UID: "54cff721-5d48-4e7a-9771-1b859bc68609") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:17:55.556531 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:55.556206 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 09:17:55.556531 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:55.556262 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret podName:a5680bfb-a669-4640-876e-84a6322b2454 nodeName:}" failed. No retries permitted until 2026-07-22 09:17:59.556247021 +0000 UTC m=+10.020041453 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret") pod "global-pull-secret-syncer-49l4x" (UID: "a5680bfb-a669-4640-876e-84a6322b2454") : object "kube-system"/"original-pull-secret" not registered Jul 22 09:17:55.657659 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:55.656923 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvhx\" (UniqueName: \"kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx\") pod \"network-check-target-mrw5j\" (UID: \"2e590737-a6b2-482a-9cf2-23d1add97650\") " pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:17:55.657659 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:55.657159 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 09:17:55.657659 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:55.657182 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 09:17:55.657659 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:55.657196 2549 projected.go:196] Error preparing data for projected volume kube-api-access-5zvhx for pod openshift-network-diagnostics/network-check-target-mrw5j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:17:55.657659 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:55.657255 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx podName:2e590737-a6b2-482a-9cf2-23d1add97650 nodeName:}" failed. No retries permitted until 2026-07-22 09:17:59.657236901 +0000 UTC m=+10.121031339 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5zvhx" (UniqueName: "kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx") pod "network-check-target-mrw5j" (UID: "2e590737-a6b2-482a-9cf2-23d1add97650") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:17:57.334654 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:57.333744 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:17:57.334654 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:57.333879 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6sjqx" podUID="54cff721-5d48-4e7a-9771-1b859bc68609" Jul 22 09:17:57.334654 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:57.334380 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:17:57.334654 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:57.334478 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrw5j" podUID="2e590737-a6b2-482a-9cf2-23d1add97650" Jul 22 09:17:57.334654 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:57.334539 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:17:57.334654 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:57.334614 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49l4x" podUID="a5680bfb-a669-4640-876e-84a6322b2454" Jul 22 09:17:59.334255 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:59.333777 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:17:59.334255 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:59.333914 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49l4x" podUID="a5680bfb-a669-4640-876e-84a6322b2454" Jul 22 09:17:59.334255 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:59.333951 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:17:59.334255 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:59.333986 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:17:59.334255 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:59.334086 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrw5j" podUID="2e590737-a6b2-482a-9cf2-23d1add97650" Jul 22 09:17:59.334255 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:59.334182 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6sjqx" podUID="54cff721-5d48-4e7a-9771-1b859bc68609" Jul 22 09:17:59.592101 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:59.592019 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs\") pod \"network-metrics-daemon-6sjqx\" (UID: \"54cff721-5d48-4e7a-9771-1b859bc68609\") " pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:17:59.592101 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:59.592091 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret\") pod \"global-pull-secret-syncer-49l4x\" (UID: \"a5680bfb-a669-4640-876e-84a6322b2454\") " pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:17:59.592340 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:59.592144 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:17:59.592340 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:59.592215 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs podName:54cff721-5d48-4e7a-9771-1b859bc68609 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:07.592195818 +0000 UTC m=+18.055990248 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs") pod "network-metrics-daemon-6sjqx" (UID: "54cff721-5d48-4e7a-9771-1b859bc68609") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:17:59.592340 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:59.592214 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 09:17:59.592340 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:59.592253 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret podName:a5680bfb-a669-4640-876e-84a6322b2454 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:07.592242679 +0000 UTC m=+18.056037110 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret") pod "global-pull-secret-syncer-49l4x" (UID: "a5680bfb-a669-4640-876e-84a6322b2454") : object "kube-system"/"original-pull-secret" not registered Jul 22 09:17:59.693293 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:17:59.693223 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvhx\" (UniqueName: \"kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx\") pod \"network-check-target-mrw5j\" (UID: \"2e590737-a6b2-482a-9cf2-23d1add97650\") " pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:17:59.693487 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:59.693450 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 09:17:59.693487 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:59.693474 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 09:17:59.693487 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:59.693486 2549 projected.go:196] Error preparing data for projected volume kube-api-access-5zvhx for pod openshift-network-diagnostics/network-check-target-mrw5j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:17:59.693639 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:17:59.693544 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx podName:2e590737-a6b2-482a-9cf2-23d1add97650 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:07.69352399 +0000 UTC m=+18.157318426 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5zvhx" (UniqueName: "kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx") pod "network-check-target-mrw5j" (UID: "2e590737-a6b2-482a-9cf2-23d1add97650") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:18:01.333894 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:01.333838 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:18:01.334356 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:01.333871 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:18:01.334356 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:01.333992 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:18:01.334356 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:01.333992 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrw5j" podUID="2e590737-a6b2-482a-9cf2-23d1add97650" Jul 22 09:18:01.334356 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:01.334106 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49l4x" podUID="a5680bfb-a669-4640-876e-84a6322b2454" Jul 22 09:18:01.334356 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:01.334224 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6sjqx" podUID="54cff721-5d48-4e7a-9771-1b859bc68609" Jul 22 09:18:03.333763 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:03.333726 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:18:03.334138 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:03.333726 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:18:03.334138 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:03.333836 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49l4x" podUID="a5680bfb-a669-4640-876e-84a6322b2454" Jul 22 09:18:03.334138 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:03.333736 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:18:03.334138 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:03.333951 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6sjqx" podUID="54cff721-5d48-4e7a-9771-1b859bc68609" Jul 22 09:18:03.334138 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:03.334041 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrw5j" podUID="2e590737-a6b2-482a-9cf2-23d1add97650" Jul 22 09:18:05.333689 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:05.333654 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:18:05.334110 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:05.333654 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:18:05.334110 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:05.333800 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6sjqx" podUID="54cff721-5d48-4e7a-9771-1b859bc68609" Jul 22 09:18:05.334110 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:05.333653 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:18:05.334110 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:05.333862 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrw5j" podUID="2e590737-a6b2-482a-9cf2-23d1add97650" Jul 22 09:18:05.334110 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:05.333953 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49l4x" podUID="a5680bfb-a669-4640-876e-84a6322b2454" Jul 22 09:18:07.333945 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:07.333910 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:18:07.334398 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:07.333909 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:18:07.334398 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:07.334047 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49l4x" podUID="a5680bfb-a669-4640-876e-84a6322b2454" Jul 22 09:18:07.334398 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:07.333909 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:18:07.334398 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:07.334141 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrw5j" podUID="2e590737-a6b2-482a-9cf2-23d1add97650" Jul 22 09:18:07.334398 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:07.334224 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6sjqx" podUID="54cff721-5d48-4e7a-9771-1b859bc68609" Jul 22 09:18:07.651615 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:07.651532 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret\") pod \"global-pull-secret-syncer-49l4x\" (UID: \"a5680bfb-a669-4640-876e-84a6322b2454\") " pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:18:07.651615 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:07.651598 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs\") pod \"network-metrics-daemon-6sjqx\" (UID: \"54cff721-5d48-4e7a-9771-1b859bc68609\") " pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:18:07.651781 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:07.651689 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:18:07.651781 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:07.651696 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 09:18:07.651781 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:07.651741 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs podName:54cff721-5d48-4e7a-9771-1b859bc68609 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:23.651727289 +0000 UTC m=+34.115521718 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs") pod "network-metrics-daemon-6sjqx" (UID: "54cff721-5d48-4e7a-9771-1b859bc68609") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:18:07.651781 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:07.651754 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret podName:a5680bfb-a669-4640-876e-84a6322b2454 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:23.651748094 +0000 UTC m=+34.115542510 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret") pod "global-pull-secret-syncer-49l4x" (UID: "a5680bfb-a669-4640-876e-84a6322b2454") : object "kube-system"/"original-pull-secret" not registered Jul 22 09:18:07.752835 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:07.752798 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvhx\" (UniqueName: \"kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx\") pod \"network-check-target-mrw5j\" (UID: \"2e590737-a6b2-482a-9cf2-23d1add97650\") " pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:18:07.753001 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:07.752982 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 09:18:07.753042 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:07.753008 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 09:18:07.753042 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:07.753022 2549 projected.go:196] Error preparing data for projected volume kube-api-access-5zvhx for pod openshift-network-diagnostics/network-check-target-mrw5j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:18:07.753120 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:07.753092 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx podName:2e590737-a6b2-482a-9cf2-23d1add97650 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:23.753072377 +0000 UTC m=+34.216866793 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5zvhx" (UniqueName: "kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx") pod "network-check-target-mrw5j" (UID: "2e590737-a6b2-482a-9cf2-23d1add97650") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:18:09.333605 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:09.333569 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:18:09.334046 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:09.333580 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:18:09.334046 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:09.333707 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6sjqx" podUID="54cff721-5d48-4e7a-9771-1b859bc68609" Jul 22 09:18:09.334046 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:09.333580 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:18:09.334046 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:09.333785 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49l4x" podUID="a5680bfb-a669-4640-876e-84a6322b2454" Jul 22 09:18:09.334046 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:09.333852 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrw5j" podUID="2e590737-a6b2-482a-9cf2-23d1add97650" Jul 22 09:18:11.334178 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:11.334136 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:18:11.334621 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:11.334183 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:18:11.334621 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:11.334136 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:18:11.334621 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:11.334268 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6sjqx" podUID="54cff721-5d48-4e7a-9771-1b859bc68609" Jul 22 09:18:11.334621 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:11.334326 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49l4x" podUID="a5680bfb-a669-4640-876e-84a6322b2454" Jul 22 09:18:11.334621 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:11.334379 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrw5j" podUID="2e590737-a6b2-482a-9cf2-23d1add97650" Jul 22 09:18:13.334097 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.333887 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:18:13.334991 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.333891 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:18:13.334991 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.333923 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:18:13.334991 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:13.334218 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49l4x" podUID="a5680bfb-a669-4640-876e-84a6322b2454" Jul 22 09:18:13.334991 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:13.334320 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6sjqx" podUID="54cff721-5d48-4e7a-9771-1b859bc68609" Jul 22 09:18:13.334991 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:13.334382 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrw5j" podUID="2e590737-a6b2-482a-9cf2-23d1add97650" Jul 22 09:18:13.432195 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.432164 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" event={"ID":"7dd0f078-e04e-41fb-925d-09f6e234e4b5","Type":"ContainerStarted","Data":"3fb38696f85786ea58116f388c5199808113927017a09fbc95460ffe11d7e769"} Jul 22 09:18:13.434987 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.434969 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcsgw_dc8d8789-74a0-44ec-8902-44a9650c41ac/ovn-acl-logging/0.log" Jul 22 09:18:13.435296 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.435277 2549 generic.go:356] "Generic (PLEG): container finished" podID="dc8d8789-74a0-44ec-8902-44a9650c41ac" containerID="94920394c7bc9ca12ab11d9dc3c1fc0611fdfb2883f53f2a9a1c24b78d9b60fa" exitCode=1 Jul 22 09:18:13.435360 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.435339 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" event={"ID":"dc8d8789-74a0-44ec-8902-44a9650c41ac","Type":"ContainerStarted","Data":"a1d05651d9822de55a6ea0834f7ac501cceb97586ceacb7244a9d10126bbaef2"} Jul 22 09:18:13.435422 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.435407 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" event={"ID":"dc8d8789-74a0-44ec-8902-44a9650c41ac","Type":"ContainerStarted","Data":"4e5afd701160d2c7ca9ec153dca261d6aefdad16fee0e6aff58f561e56c3dc67"} Jul 22 09:18:13.435476 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.435454 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" event={"ID":"dc8d8789-74a0-44ec-8902-44a9650c41ac","Type":"ContainerStarted","Data":"bccfa4c068f04e471aa47f3e30edb8bbf992bcc47ad5cc72be601171bb1195a1"} Jul 22 09:18:13.435527 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.435493 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" event={"ID":"dc8d8789-74a0-44ec-8902-44a9650c41ac","Type":"ContainerStarted","Data":"81c430e7edd4c674cf592827fa47f223c4e4abb5f72db999d2cd41dc5415b643"} Jul 22 09:18:13.435527 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.435522 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" event={"ID":"dc8d8789-74a0-44ec-8902-44a9650c41ac","Type":"ContainerDied","Data":"94920394c7bc9ca12ab11d9dc3c1fc0611fdfb2883f53f2a9a1c24b78d9b60fa"} Jul 22 09:18:13.435605 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.435549 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" event={"ID":"dc8d8789-74a0-44ec-8902-44a9650c41ac","Type":"ContainerStarted","Data":"ae3dd9bdbe62623649eeb3c29e1ab81a6fad0fb0db8153c734f6b69520ef477e"} Jul 22 09:18:13.439398 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.439372 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" event={"ID":"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f","Type":"ContainerStarted","Data":"bacedcd90a59c11179c5a6e6b0ec5f8ebbbb66e30667540e03dd906ea965f53d"} Jul 22 09:18:13.440727 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.440707 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7lqtp" event={"ID":"f757fe69-cbfe-4254-85cf-07a10840f85b","Type":"ContainerStarted","Data":"3d248593497a33050a9f950b6357baa792c6cf430956227d2d53832ccf28139e"} Jul 22 09:18:13.442036 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.442016 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jrpbc" event={"ID":"9ed89b08-ed62-4f3b-96b1-3bdba846b4f3","Type":"ContainerStarted","Data":"6766ebccd26172556cb259c6dc33159c46bf7225ddae2d3be94529316fec5249"} Jul 22 09:18:13.443281 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.443255 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-659hp" event={"ID":"a87cdd8f-1617-4f31-85ab-6ba3294ef8be","Type":"ContainerStarted","Data":"6f1b2be9c9f4a86848c87537a39cbe36fc21c0479d65052e479fd533f7f63b64"} Jul 22 09:18:13.444614 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.444594 2549 generic.go:356] "Generic (PLEG): container finished" podID="f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6" containerID="52dc78d3873ef70b68786f163f46f9a07b3753a2365f81a8fb8aac9f68965b57" exitCode=0 Jul 22 09:18:13.444711 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.444652 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5f2z" event={"ID":"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6","Type":"ContainerDied","Data":"52dc78d3873ef70b68786f163f46f9a07b3753a2365f81a8fb8aac9f68965b57"} Jul 22 09:18:13.446091 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.446041 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rn8j7" event={"ID":"beb97b8f-7ed2-41fc-87b5-7a0488638e16","Type":"ContainerStarted","Data":"571a142996c88ff7f92038fa2b9ede99416bb1e0ac2bf70bb8130311cfee6ec8"} Jul 22 09:18:13.447689 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.447671 2549 generic.go:356] "Generic (PLEG): container finished" podID="16c453564e70f602f26f88ff0da5d012" containerID="669e717472c7b4b3ad2564e9d68c632070ba020a522433a357756d96cdadc735" exitCode=0 Jul 22 09:18:13.447742 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.447729 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" event={"ID":"16c453564e70f602f26f88ff0da5d012","Type":"ContainerDied","Data":"669e717472c7b4b3ad2564e9d68c632070ba020a522433a357756d96cdadc735"} Jul 22 09:18:13.449100 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.449081 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-138.ec2.internal" event={"ID":"22c71e61550bcbc1af7cc7ad68e31b2b","Type":"ContainerStarted","Data":"6d0d35246d768a0d484a05adf3b1477c3161c54ca702a5670c642b4a8a267163"} Jul 22 09:18:13.453501 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.453461 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f2mdv" podStartSLOduration=3.781914194 podStartE2EDuration="23.453450397s" totalImagesPullingTime="19.671536203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:17:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:13.45306092 +0000 UTC m=+23.916855359" watchObservedRunningTime="2026-07-22 09:18:13.453450397 +0000 UTC m=+23.917244834" Jul 22 09:18:13.466244 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.466211 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-138.ec2.internal" podStartSLOduration=22.466198445 podStartE2EDuration="22.466198445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:13.465892629 +0000 UTC m=+23.929687067" watchObservedRunningTime="2026-07-22 09:18:13.466198445 +0000 UTC m=+23.929992883" Jul 22 09:18:13.481169 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.481130 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rn8j7" podStartSLOduration=8.479391271 podStartE2EDuration="23.48111413s" totalImagesPullingTime="15.001722859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:17:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:13.48105535 +0000 UTC m=+23.944849789" watchObservedRunningTime="2026-07-22 09:18:13.48111413 +0000 UTC m=+23.944908569" Jul 22 09:18:13.517048 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.516990 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7lqtp" podStartSLOduration=3.835649872 podStartE2EDuration="23.516972013s" totalImagesPullingTime="19.681322141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:17:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:13.516245022 +0000 UTC m=+23.980039455" watchObservedRunningTime="2026-07-22 09:18:13.516972013 +0000 UTC m=+23.980766452" Jul 22 09:18:13.562139 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.562109 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:13.564027 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.563980 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jrpbc" podStartSLOduration=3.3197662230000002 podStartE2EDuration="23.56396036s" totalImagesPullingTime="20.244194137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:17:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:13.549960609 +0000 UTC m=+24.013755047" watchObservedRunningTime="2026-07-22 09:18:13.56396036 +0000 UTC m=+24.027754795" Jul 22 09:18:13.564589 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.564556 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-659hp" podStartSLOduration=4.224377756 podStartE2EDuration="23.564548095s" totalImagesPullingTime="19.340170339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:17:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:13.564105887 +0000 UTC m=+24.027900326" watchObservedRunningTime="2026-07-22 09:18:13.564548095 +0000 UTC m=+24.028342534" Jul 22 09:18:13.674819 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.674771 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:13.774732 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.774620 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:13.845794 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:13.845757 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:14.328072 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:14.328013 2549 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 09:18:14.456221 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:14.456182 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" event={"ID":"16c453564e70f602f26f88ff0da5d012","Type":"ContainerStarted","Data":"965be93432f67e7b619622c4f6abed3e04c4ab7c1e1583f957b65d6083d10c5a"} Jul 22 09:18:14.457639 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:14.457609 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z9599" event={"ID":"af47e4da-92fc-4f12-abcf-c4c2975d28ba","Type":"ContainerStarted","Data":"52a6fdf8accde4d936db6a5a997caec5d5081a6739ee376b03cf9c6737a918cd"} Jul 22 09:18:14.459292 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:14.459265 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" event={"ID":"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f","Type":"ContainerStarted","Data":"92af5b6492d210e9af1a2228f32f65770120b51f4f5132e5de86b36037443773"} Jul 22 09:18:14.473092 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:14.473051 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-138.ec2.internal" podStartSLOduration=23.473038451 podStartE2EDuration="23.473038451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:14.472713815 +0000 UTC m=+24.936508254" watchObservedRunningTime="2026-07-22 09:18:14.473038451 +0000 UTC m=+24.936832889" Jul 22 09:18:14.488847 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:14.488799 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-z9599" podStartSLOduration=4.794982836 podStartE2EDuration="24.488784628s" totalImagesPullingTime="19.693801792s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:17:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:14.488422877 +0000 UTC m=+24.952217315" watchObservedRunningTime="2026-07-22 09:18:14.488784628 +0000 UTC m=+24.952579066" Jul 22 09:18:14.615411 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:14.615325 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:14.780844 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:14.780812 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7lqtp" Jul 22 09:18:14.781605 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:14.781582 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7lqtp" Jul 22 09:18:15.280139 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:15.280046 2549 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T09:18:14.328031582Z","UUID":"1c890613-a9a5-4d0d-9bb8-7e3580389bd2","Handler":null,"Name":"","Endpoint":""} Jul 22 09:18:15.281386 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:15.281366 2549 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 09:18:15.281459 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:15.281402 2549 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 09:18:15.334319 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:15.334276 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:18:15.334479 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:15.334276 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:18:15.334479 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:15.334390 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6sjqx" podUID="54cff721-5d48-4e7a-9771-1b859bc68609" Jul 22 09:18:15.334479 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:15.334455 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrw5j" podUID="2e590737-a6b2-482a-9cf2-23d1add97650" Jul 22 09:18:15.334479 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:15.334276 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:18:15.334638 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:15.334552 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49l4x" podUID="a5680bfb-a669-4640-876e-84a6322b2454" Jul 22 09:18:15.461479 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:15.461443 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7lqtp" Jul 22 09:18:15.461859 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:15.461676 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7lqtp" Jul 22 09:18:16.467051 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:16.467025 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcsgw_dc8d8789-74a0-44ec-8902-44a9650c41ac/ovn-acl-logging/0.log" Jul 22 09:18:16.467679 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:16.467458 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" event={"ID":"dc8d8789-74a0-44ec-8902-44a9650c41ac","Type":"ContainerStarted","Data":"3750687e26cdd6f5fe6217aa6f00b1c3bce56db5f28034c8eee6a7f5fb37b86e"} Jul 22 09:18:16.469776 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:16.469739 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" event={"ID":"fdd9efb5-2c5e-4810-936b-d4c29fe3cd1f","Type":"ContainerStarted","Data":"c71407fba59c45cbace559c4d0014198d29615ed69a5dfb3ec707b3925151e04"} Jul 22 09:18:17.067002 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:17.066962 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:17.146088 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:17.146056 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:17.334288 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:17.334204 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:18:17.334288 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:17.334240 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:18:17.334621 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:17.334204 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:18:17.334621 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:17.334340 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49l4x" podUID="a5680bfb-a669-4640-876e-84a6322b2454" Jul 22 09:18:17.334621 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:17.334390 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrw5j" podUID="2e590737-a6b2-482a-9cf2-23d1add97650" Jul 22 09:18:17.334621 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:17.334481 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6sjqx" podUID="54cff721-5d48-4e7a-9771-1b859bc68609" Jul 22 09:18:18.395067 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:18.394866 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:18.478379 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:18.478349 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcsgw_dc8d8789-74a0-44ec-8902-44a9650c41ac/ovn-acl-logging/0.log" Jul 22 09:18:18.478787 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:18.478762 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" event={"ID":"dc8d8789-74a0-44ec-8902-44a9650c41ac","Type":"ContainerStarted","Data":"964c0d1a7296d4a89993a1732988aa04c45eb9c3c7a058b5a51ddb5ef1684ce1"} Jul 22 09:18:18.479445 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:18.479191 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:18:18.479445 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:18.479271 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:18:18.479445 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:18.479343 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:18:18.479445 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:18.479369 2549 scope.go:117] "RemoveContainer" containerID="94920394c7bc9ca12ab11d9dc3c1fc0611fdfb2883f53f2a9a1c24b78d9b60fa" Jul 22 09:18:18.495465 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:18.495441 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:18:18.495586 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:18.495575 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:18:18.501114 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:18.501088 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:18.519240 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:18.519191 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxzpg" podStartSLOduration=6.612181234 podStartE2EDuration="28.519174812s" totalImagesPullingTime="21.906993578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:17:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:16.493726997 +0000 UTC m=+26.957521438" watchObservedRunningTime="2026-07-22 09:18:18.519174812 +0000 UTC m=+28.982969270" Jul 22 09:18:19.334392 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:19.334356 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:18:19.334392 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:19.334381 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:18:19.334591 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:19.334548 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6sjqx" podUID="54cff721-5d48-4e7a-9771-1b859bc68609" Jul 22 09:18:19.334641 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:19.334587 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:18:19.334778 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:19.334751 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49l4x" podUID="a5680bfb-a669-4640-876e-84a6322b2454" Jul 22 09:18:19.337201 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:19.334869 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrw5j" podUID="2e590737-a6b2-482a-9cf2-23d1add97650" Jul 22 09:18:19.708281 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:19.708190 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mrw5j"] Jul 22 09:18:19.709081 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:19.708342 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:18:19.709081 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:19.708443 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrw5j" podUID="2e590737-a6b2-482a-9cf2-23d1add97650" Jul 22 09:18:19.711273 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:19.711223 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-49l4x"] Jul 22 09:18:19.711421 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:19.711387 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:18:19.711508 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:19.711473 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49l4x" podUID="a5680bfb-a669-4640-876e-84a6322b2454" Jul 22 09:18:19.720953 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:19.720929 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6sjqx"] Jul 22 09:18:19.721071 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:19.721028 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:18:19.721142 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:19.721126 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6sjqx" podUID="54cff721-5d48-4e7a-9771-1b859bc68609" Jul 22 09:18:20.487586 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:20.487374 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcsgw_dc8d8789-74a0-44ec-8902-44a9650c41ac/ovn-acl-logging/0.log" Jul 22 09:18:20.487910 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:20.487888 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" event={"ID":"dc8d8789-74a0-44ec-8902-44a9650c41ac","Type":"ContainerStarted","Data":"a3cb0d3e18a330e64192b5c1f5a86485505207b1310cfc707734244249686e24"} Jul 22 09:18:20.489505 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:20.489481 2549 generic.go:356] "Generic (PLEG): container finished" podID="f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6" containerID="1f3ad3a17bbc27c6c58666bde93c686e916ad866e2ee8d3dbc0abe3c4966eb19" exitCode=0 Jul 22 09:18:20.489589 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:20.489525 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5f2z" event={"ID":"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6","Type":"ContainerDied","Data":"1f3ad3a17bbc27c6c58666bde93c686e916ad866e2ee8d3dbc0abe3c4966eb19"} Jul 22 09:18:20.517523 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:20.517478 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" podStartSLOduration=10.73534877 podStartE2EDuration="30.517464768s" totalImagesPullingTime="19.782115998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:17:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:20.517147995 +0000 UTC m=+30.980942435" watchObservedRunningTime="2026-07-22 09:18:20.517464768 +0000 UTC m=+30.981259205" Jul 22 09:18:21.334399 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:21.334355 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:18:21.334749 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:21.334422 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:18:21.334749 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:21.334522 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:18:21.334749 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:21.334538 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49l4x" podUID="a5680bfb-a669-4640-876e-84a6322b2454" Jul 22 09:18:21.334749 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:21.334617 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6sjqx" podUID="54cff721-5d48-4e7a-9771-1b859bc68609" Jul 22 09:18:21.334749 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:21.334688 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrw5j" podUID="2e590737-a6b2-482a-9cf2-23d1add97650" Jul 22 09:18:22.463656 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:22.463625 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:22.496121 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:22.496092 2549 generic.go:356] "Generic (PLEG): container finished" podID="f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6" containerID="23e2c56669f1186e3602c2bc5fa73fe8d0d626b91e7ce6165cfeaede3e427869" exitCode=0 Jul 22 09:18:22.496239 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:22.496129 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5f2z" event={"ID":"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6","Type":"ContainerDied","Data":"23e2c56669f1186e3602c2bc5fa73fe8d0d626b91e7ce6165cfeaede3e427869"} Jul 22 09:18:22.551227 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:22.551150 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:23.333982 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:23.333944 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:18:23.334145 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:23.334045 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:18:23.334145 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:23.334052 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrw5j" podUID="2e590737-a6b2-482a-9cf2-23d1add97650" Jul 22 09:18:23.334145 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:23.334077 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:18:23.334345 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:23.334171 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49l4x" podUID="a5680bfb-a669-4640-876e-84a6322b2454" Jul 22 09:18:23.334345 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:23.334268 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6sjqx" podUID="54cff721-5d48-4e7a-9771-1b859bc68609" Jul 22 09:18:23.501356 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:23.501245 2549 generic.go:356] "Generic (PLEG): container finished" podID="f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6" containerID="be4123a1af8b4ec33215225b173a65edc86b72f5f7bad0be0f38dd87f98a7166" exitCode=0 Jul 22 09:18:23.501356 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:23.501320 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5f2z" event={"ID":"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6","Type":"ContainerDied","Data":"be4123a1af8b4ec33215225b173a65edc86b72f5f7bad0be0f38dd87f98a7166"} Jul 22 09:18:23.501813 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:23.501413 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:23.571226 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:23.571189 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:23.674541 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:23.674502 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret\") pod \"global-pull-secret-syncer-49l4x\" (UID: \"a5680bfb-a669-4640-876e-84a6322b2454\") " pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:18:23.674697 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:23.674591 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs\") pod \"network-metrics-daemon-6sjqx\" (UID: \"54cff721-5d48-4e7a-9771-1b859bc68609\") " pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:18:23.674758 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:23.674707 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 09:18:23.674808 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:23.674771 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:18:23.674808 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:23.674783 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret podName:a5680bfb-a669-4640-876e-84a6322b2454 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:55.674762358 +0000 UTC m=+66.138556796 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret") pod "global-pull-secret-syncer-49l4x" (UID: "a5680bfb-a669-4640-876e-84a6322b2454") : object "kube-system"/"original-pull-secret" not registered Jul 22 09:18:23.674911 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:23.674826 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs podName:54cff721-5d48-4e7a-9771-1b859bc68609 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:55.674813759 +0000 UTC m=+66.138608184 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs") pod "network-metrics-daemon-6sjqx" (UID: "54cff721-5d48-4e7a-9771-1b859bc68609") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:18:23.775107 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:23.775020 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvhx\" (UniqueName: \"kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx\") pod \"network-check-target-mrw5j\" (UID: \"2e590737-a6b2-482a-9cf2-23d1add97650\") " pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:18:23.775237 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:23.775174 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 09:18:23.775237 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:23.775195 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 09:18:23.775237 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:23.775204 2549 projected.go:196] Error preparing data for projected volume kube-api-access-5zvhx for pod openshift-network-diagnostics/network-check-target-mrw5j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:18:23.775382 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:23.775253 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx podName:2e590737-a6b2-482a-9cf2-23d1add97650 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:55.775239681 +0000 UTC m=+66.239034111 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5zvhx" (UniqueName: "kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx") pod "network-check-target-mrw5j" (UID: "2e590737-a6b2-482a-9cf2-23d1add97650") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:18:23.966827 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:23.966775 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:24.066365 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:24.066332 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:24.240073 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:24.240040 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:24.343502 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:24.343418 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:25.023816 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.023780 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:25.222031 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.221999 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:25.281750 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.281713 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:25.333801 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.333773 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:18:25.333801 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.333783 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:18:25.334029 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.333899 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:18:25.334029 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:25.333912 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6sjqx" podUID="54cff721-5d48-4e7a-9771-1b859bc68609" Jul 22 09:18:25.334029 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:25.334012 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49l4x" podUID="a5680bfb-a669-4640-876e-84a6322b2454" Jul 22 09:18:25.334171 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:25.334095 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrw5j" podUID="2e590737-a6b2-482a-9cf2-23d1add97650" Jul 22 09:18:25.844799 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.844745 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:25.879282 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.879255 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-138.ec2.internal" event="NodeReady" Jul 22 09:18:25.879494 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.879480 2549 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 09:18:25.919021 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.918986 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d744c845d-jlz5w"] Jul 22 09:18:25.923434 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.923407 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:25.926086 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.925909 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 22 09:18:25.926086 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.926016 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 22 09:18:25.926448 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.926429 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dm8hb\"" Jul 22 09:18:25.926533 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.926454 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 22 09:18:25.929636 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.929612 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:25.931388 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.931365 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 22 09:18:25.932602 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.932554 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wzwlh"] Jul 22 09:18:25.937081 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.937063 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xrlw2"] Jul 22 09:18:25.937208 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.937192 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wzwlh" Jul 22 09:18:25.942394 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.941782 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d744c845d-jlz5w"] Jul 22 09:18:25.942394 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.941829 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:25.942549 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.942490 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 09:18:25.942600 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.942570 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-v4ns8\"" Jul 22 09:18:25.942696 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.942678 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 09:18:25.942878 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.942855 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 09:18:25.944401 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.944378 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 09:18:25.944490 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.944400 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7wntr\"" Jul 22 09:18:25.944672 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.944652 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 09:18:25.944672 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.944664 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 09:18:25.944814 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.944791 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 09:18:25.948710 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.948693 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xrlw2"] Jul 22 09:18:25.956172 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.956141 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wzwlh"] Jul 22 09:18:25.956727 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.956706 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-k4mr8"] Jul 22 09:18:25.960650 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.960633 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k4mr8" Jul 22 09:18:25.962888 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.962868 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 09:18:25.962969 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.962878 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z2clq\"" Jul 22 09:18:25.962969 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.962878 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 09:18:25.969933 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.969915 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k4mr8"] Jul 22 09:18:25.970939 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:25.970909 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-659hp_a87cdd8f-1617-4f31-85ab-6ba3294ef8be/dns-node-resolver/0.log" Jul 22 09:18:26.095231 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.095146 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-bound-sa-token\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.095231 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.095182 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44d91ea8-9844-4ae9-9af8-b5297d3e1639-tmp-dir\") pod \"dns-default-k4mr8\" (UID: \"44d91ea8-9844-4ae9-9af8-b5297d3e1639\") " pod="openshift-dns/dns-default-k4mr8" Jul 22 09:18:26.095231 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.095208 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44d91ea8-9844-4ae9-9af8-b5297d3e1639-metrics-tls\") pod \"dns-default-k4mr8\" (UID: \"44d91ea8-9844-4ae9-9af8-b5297d3e1639\") " pod="openshift-dns/dns-default-k4mr8" Jul 22 09:18:26.095735 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.095251 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f40f3b1a-129d-42b5-8b2d-99e272295571-cert\") pod \"ingress-canary-wzwlh\" (UID: \"f40f3b1a-129d-42b5-8b2d-99e272295571\") " pod="openshift-ingress-canary/ingress-canary-wzwlh" Jul 22 09:18:26.095735 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.095282 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-tls\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.095735 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.095358 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/263ef12e-19a9-47c9-adc5-e8b0ec511533-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:26.095735 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.095395 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ae4a46e-063d-495a-87c2-f571c93e4850-ca-trust-extracted\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.095735 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.095520 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/263ef12e-19a9-47c9-adc5-e8b0ec511533-data-volume\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:26.095735 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.095565 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhbrq\" (UniqueName: \"kubernetes.io/projected/44d91ea8-9844-4ae9-9af8-b5297d3e1639-kube-api-access-vhbrq\") pod \"dns-default-k4mr8\" (UID: \"44d91ea8-9844-4ae9-9af8-b5297d3e1639\") " pod="openshift-dns/dns-default-k4mr8" Jul 22 09:18:26.095735 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.095590 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp6bx\" (UniqueName: \"kubernetes.io/projected/263ef12e-19a9-47c9-adc5-e8b0ec511533-kube-api-access-bp6bx\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:26.095735 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.095645 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/263ef12e-19a9-47c9-adc5-e8b0ec511533-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:26.095735 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.095671 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ae4a46e-063d-495a-87c2-f571c93e4850-image-registry-private-configuration\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.095735 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.095702 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88dql\" (UniqueName: \"kubernetes.io/projected/f40f3b1a-129d-42b5-8b2d-99e272295571-kube-api-access-88dql\") pod \"ingress-canary-wzwlh\" (UID: \"f40f3b1a-129d-42b5-8b2d-99e272295571\") " pod="openshift-ingress-canary/ingress-canary-wzwlh" Jul 22 09:18:26.095735 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.095728 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-certificates\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.096108 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.095751 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnw66\" (UniqueName: \"kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-kube-api-access-jnw66\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.096108 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.095783 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ae4a46e-063d-495a-87c2-f571c93e4850-trusted-ca\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.096108 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.095824 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/263ef12e-19a9-47c9-adc5-e8b0ec511533-crio-socket\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:26.096108 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.095855 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/44d91ea8-9844-4ae9-9af8-b5297d3e1639-config-volume\") pod \"dns-default-k4mr8\" (UID: \"44d91ea8-9844-4ae9-9af8-b5297d3e1639\") " pod="openshift-dns/dns-default-k4mr8" Jul 22 09:18:26.096108 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.095884 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ae4a46e-063d-495a-87c2-f571c93e4850-installation-pull-secrets\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.100290 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.100263 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:26.129508 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.129483 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-5982h"] Jul 22 09:18:26.133263 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.133241 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-5982h" Jul 22 09:18:26.135795 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.135771 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vhwp4\"" Jul 22 09:18:26.135896 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.135771 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 22 09:18:26.136842 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.136821 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 22 09:18:26.145339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.145298 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-5982h"] Jul 22 09:18:26.182253 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.182207 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:26.196754 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.196726 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/263ef12e-19a9-47c9-adc5-e8b0ec511533-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:26.196884 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.196773 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ae4a46e-063d-495a-87c2-f571c93e4850-ca-trust-extracted\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.196884 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.196819 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/263ef12e-19a9-47c9-adc5-e8b0ec511533-data-volume\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:26.196884 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.196848 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhbrq\" (UniqueName: \"kubernetes.io/projected/44d91ea8-9844-4ae9-9af8-b5297d3e1639-kube-api-access-vhbrq\") pod \"dns-default-k4mr8\" (UID: \"44d91ea8-9844-4ae9-9af8-b5297d3e1639\") " pod="openshift-dns/dns-default-k4mr8" Jul 22 09:18:26.196884 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.196872 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp6bx\" (UniqueName: \"kubernetes.io/projected/263ef12e-19a9-47c9-adc5-e8b0ec511533-kube-api-access-bp6bx\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:26.197070 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.196909 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/263ef12e-19a9-47c9-adc5-e8b0ec511533-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:26.197070 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.196932 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ae4a46e-063d-495a-87c2-f571c93e4850-image-registry-private-configuration\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.197070 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.196961 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88dql\" (UniqueName: \"kubernetes.io/projected/f40f3b1a-129d-42b5-8b2d-99e272295571-kube-api-access-88dql\") pod \"ingress-canary-wzwlh\" (UID: \"f40f3b1a-129d-42b5-8b2d-99e272295571\") " pod="openshift-ingress-canary/ingress-canary-wzwlh" Jul 22 09:18:26.197070 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:26.196876 2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 09:18:26.197070 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:26.196996 2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 09:18:26.197070 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:26.197053 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/263ef12e-19a9-47c9-adc5-e8b0ec511533-kube-rbac-proxy-cm podName:263ef12e-19a9-47c9-adc5-e8b0ec511533 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:26.697034424 +0000 UTC m=+37.160828855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/263ef12e-19a9-47c9-adc5-e8b0ec511533-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xrlw2" (UID: "263ef12e-19a9-47c9-adc5-e8b0ec511533") : configmap "kube-rbac-proxy" not found Jul 22 09:18:26.197364 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.197074 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-certificates\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.197364 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.197103 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnw66\" (UniqueName: \"kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-kube-api-access-jnw66\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.197364 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:26.197125 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/263ef12e-19a9-47c9-adc5-e8b0ec511533-insights-runtime-extractor-tls podName:263ef12e-19a9-47c9-adc5-e8b0ec511533 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:26.697115843 +0000 UTC m=+37.160910258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/263ef12e-19a9-47c9-adc5-e8b0ec511533-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xrlw2" (UID: "263ef12e-19a9-47c9-adc5-e8b0ec511533") : secret "insights-runtime-extractor-tls" not found Jul 22 09:18:26.197364 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.197157 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ae4a46e-063d-495a-87c2-f571c93e4850-trusted-ca\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.197364 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.197199 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/263ef12e-19a9-47c9-adc5-e8b0ec511533-crio-socket\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:26.197364 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.197229 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/44d91ea8-9844-4ae9-9af8-b5297d3e1639-config-volume\") pod \"dns-default-k4mr8\" (UID: \"44d91ea8-9844-4ae9-9af8-b5297d3e1639\") " pod="openshift-dns/dns-default-k4mr8" Jul 22 09:18:26.197364 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.197257 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ae4a46e-063d-495a-87c2-f571c93e4850-installation-pull-secrets\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.197364 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.197281 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-bound-sa-token\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.197364 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.197332 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44d91ea8-9844-4ae9-9af8-b5297d3e1639-tmp-dir\") pod \"dns-default-k4mr8\" (UID: \"44d91ea8-9844-4ae9-9af8-b5297d3e1639\") " pod="openshift-dns/dns-default-k4mr8" Jul 22 09:18:26.197364 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.197359 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44d91ea8-9844-4ae9-9af8-b5297d3e1639-metrics-tls\") pod \"dns-default-k4mr8\" (UID: \"44d91ea8-9844-4ae9-9af8-b5297d3e1639\") " pod="openshift-dns/dns-default-k4mr8" Jul 22 09:18:26.197768 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.197384 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f40f3b1a-129d-42b5-8b2d-99e272295571-cert\") pod \"ingress-canary-wzwlh\" (UID: \"f40f3b1a-129d-42b5-8b2d-99e272295571\") " pod="openshift-ingress-canary/ingress-canary-wzwlh" Jul 22 09:18:26.197768 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.197412 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-tls\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.197768 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.197477 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/263ef12e-19a9-47c9-adc5-e8b0ec511533-data-volume\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:26.197768 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:26.197521 2549 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 09:18:26.197768 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:26.197534 2549 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d744c845d-jlz5w: secret "image-registry-tls" not found Jul 22 09:18:26.197768 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:26.197577 2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 09:18:26.197768 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.197579 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ae4a46e-063d-495a-87c2-f571c93e4850-ca-trust-extracted\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.197768 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:26.197581 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-tls podName:5ae4a46e-063d-495a-87c2-f571c93e4850 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:26.697565646 +0000 UTC m=+37.161360076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-tls") pod "image-registry-5d744c845d-jlz5w" (UID: "5ae4a46e-063d-495a-87c2-f571c93e4850") : secret "image-registry-tls" not found Jul 22 09:18:26.197768 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:26.197647 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44d91ea8-9844-4ae9-9af8-b5297d3e1639-metrics-tls podName:44d91ea8-9844-4ae9-9af8-b5297d3e1639 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:26.697628178 +0000 UTC m=+37.161422602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44d91ea8-9844-4ae9-9af8-b5297d3e1639-metrics-tls") pod "dns-default-k4mr8" (UID: "44d91ea8-9844-4ae9-9af8-b5297d3e1639") : secret "dns-default-metrics-tls" not found Jul 22 09:18:26.197768 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.197747 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-certificates\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.198184 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.197777 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44d91ea8-9844-4ae9-9af8-b5297d3e1639-tmp-dir\") pod \"dns-default-k4mr8\" (UID: \"44d91ea8-9844-4ae9-9af8-b5297d3e1639\") " pod="openshift-dns/dns-default-k4mr8" Jul 22 09:18:26.198184 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.197780 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/263ef12e-19a9-47c9-adc5-e8b0ec511533-crio-socket\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:26.198184 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.198056 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ae4a46e-063d-495a-87c2-f571c93e4850-trusted-ca\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.198600 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.198415 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/44d91ea8-9844-4ae9-9af8-b5297d3e1639-config-volume\") pod \"dns-default-k4mr8\" (UID: \"44d91ea8-9844-4ae9-9af8-b5297d3e1639\") " pod="openshift-dns/dns-default-k4mr8" Jul 22 09:18:26.202194 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.202137 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ae4a46e-063d-495a-87c2-f571c93e4850-installation-pull-secrets\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.202287 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.202193 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ae4a46e-063d-495a-87c2-f571c93e4850-image-registry-private-configuration\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.202287 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.202221 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f40f3b1a-129d-42b5-8b2d-99e272295571-cert\") pod \"ingress-canary-wzwlh\" (UID: \"f40f3b1a-129d-42b5-8b2d-99e272295571\") " pod="openshift-ingress-canary/ingress-canary-wzwlh" Jul 22 09:18:26.212883 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.212844 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp6bx\" (UniqueName: \"kubernetes.io/projected/263ef12e-19a9-47c9-adc5-e8b0ec511533-kube-api-access-bp6bx\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:26.213655 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.213599 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-bound-sa-token\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.213655 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.213643 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnw66\" (UniqueName: \"kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-kube-api-access-jnw66\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.213817 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.213746 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88dql\" (UniqueName: \"kubernetes.io/projected/f40f3b1a-129d-42b5-8b2d-99e272295571-kube-api-access-88dql\") pod \"ingress-canary-wzwlh\" (UID: \"f40f3b1a-129d-42b5-8b2d-99e272295571\") " pod="openshift-ingress-canary/ingress-canary-wzwlh" Jul 22 09:18:26.215345 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.215324 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhbrq\" (UniqueName: \"kubernetes.io/projected/44d91ea8-9844-4ae9-9af8-b5297d3e1639-kube-api-access-vhbrq\") pod \"dns-default-k4mr8\" (UID: \"44d91ea8-9844-4ae9-9af8-b5297d3e1639\") " pod="openshift-dns/dns-default-k4mr8" Jul 22 09:18:26.250798 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.250763 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wzwlh" Jul 22 09:18:26.277817 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.277781 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:26.297916 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.297887 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwxlp\" (UniqueName: \"kubernetes.io/projected/19a238da-30ed-4a74-9f90-2a32f8076615-kube-api-access-jwxlp\") pod \"migrator-7cc9c7ffd7-5982h\" (UID: \"19a238da-30ed-4a74-9f90-2a32f8076615\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-5982h" Jul 22 09:18:26.398962 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.398875 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwxlp\" (UniqueName: \"kubernetes.io/projected/19a238da-30ed-4a74-9f90-2a32f8076615-kube-api-access-jwxlp\") pod \"migrator-7cc9c7ffd7-5982h\" (UID: \"19a238da-30ed-4a74-9f90-2a32f8076615\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-5982h" Jul 22 09:18:26.410541 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.410513 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwxlp\" (UniqueName: \"kubernetes.io/projected/19a238da-30ed-4a74-9f90-2a32f8076615-kube-api-access-jwxlp\") pod \"migrator-7cc9c7ffd7-5982h\" (UID: \"19a238da-30ed-4a74-9f90-2a32f8076615\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-5982h" Jul 22 09:18:26.443796 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.443768 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-5982h" Jul 22 09:18:26.701937 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.701858 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/263ef12e-19a9-47c9-adc5-e8b0ec511533-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:26.702095 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.701936 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/263ef12e-19a9-47c9-adc5-e8b0ec511533-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:26.702095 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.701999 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44d91ea8-9844-4ae9-9af8-b5297d3e1639-metrics-tls\") pod \"dns-default-k4mr8\" (UID: \"44d91ea8-9844-4ae9-9af8-b5297d3e1639\") " pod="openshift-dns/dns-default-k4mr8" Jul 22 09:18:26.702095 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.702023 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-tls\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:26.702095 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:26.702035 2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 09:18:26.702095 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:26.702056 2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 09:18:26.702325 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:26.702104 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/263ef12e-19a9-47c9-adc5-e8b0ec511533-insights-runtime-extractor-tls podName:263ef12e-19a9-47c9-adc5-e8b0ec511533 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:27.702083479 +0000 UTC m=+38.165877925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/263ef12e-19a9-47c9-adc5-e8b0ec511533-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xrlw2" (UID: "263ef12e-19a9-47c9-adc5-e8b0ec511533") : secret "insights-runtime-extractor-tls" not found Jul 22 09:18:26.702325 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:26.702123 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/263ef12e-19a9-47c9-adc5-e8b0ec511533-kube-rbac-proxy-cm podName:263ef12e-19a9-47c9-adc5-e8b0ec511533 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:27.70211454 +0000 UTC m=+38.165908956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/263ef12e-19a9-47c9-adc5-e8b0ec511533-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xrlw2" (UID: "263ef12e-19a9-47c9-adc5-e8b0ec511533") : configmap "kube-rbac-proxy" not found Jul 22 09:18:26.702325 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:26.702123 2549 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 09:18:26.702325 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:26.702141 2549 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d744c845d-jlz5w: secret "image-registry-tls" not found Jul 22 09:18:26.702325 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:26.702158 2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 09:18:26.702325 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:26.702194 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-tls podName:5ae4a46e-063d-495a-87c2-f571c93e4850 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:27.70217837 +0000 UTC m=+38.165972809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-tls") pod "image-registry-5d744c845d-jlz5w" (UID: "5ae4a46e-063d-495a-87c2-f571c93e4850") : secret "image-registry-tls" not found Jul 22 09:18:26.702325 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:26.702220 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44d91ea8-9844-4ae9-9af8-b5297d3e1639-metrics-tls podName:44d91ea8-9844-4ae9-9af8-b5297d3e1639 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:27.702203332 +0000 UTC m=+38.165997765 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44d91ea8-9844-4ae9-9af8-b5297d3e1639-metrics-tls") pod "dns-default-k4mr8" (UID: "44d91ea8-9844-4ae9-9af8-b5297d3e1639") : secret "dns-default-metrics-tls" not found Jul 22 09:18:26.954921 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:26.954810 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:27.176316 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:27.176274 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rn8j7_beb97b8f-7ed2-41fc-87b5-7a0488638e16/node-ca/0.log" Jul 22 09:18:27.333643 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:27.333610 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:18:27.333643 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:27.333646 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:18:27.333883 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:27.333651 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:18:27.337659 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:27.337636 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 09:18:27.337783 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:27.337680 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 09:18:27.337783 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:27.337720 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8b4k6\"" Jul 22 09:18:27.338242 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:27.338219 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 09:18:27.338369 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:27.338278 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 09:18:27.338369 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:27.338295 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6sgn5\"" Jul 22 09:18:27.710483 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:27.710000 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44d91ea8-9844-4ae9-9af8-b5297d3e1639-metrics-tls\") pod \"dns-default-k4mr8\" (UID: \"44d91ea8-9844-4ae9-9af8-b5297d3e1639\") " pod="openshift-dns/dns-default-k4mr8" Jul 22 09:18:27.710483 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:27.710049 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-tls\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:27.710483 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:27.710090 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/263ef12e-19a9-47c9-adc5-e8b0ec511533-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:27.710483 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:27.710149 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/263ef12e-19a9-47c9-adc5-e8b0ec511533-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:27.710483 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:27.710247 2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 09:18:27.710483 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:27.710340 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/263ef12e-19a9-47c9-adc5-e8b0ec511533-kube-rbac-proxy-cm podName:263ef12e-19a9-47c9-adc5-e8b0ec511533 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:29.71028964 +0000 UTC m=+40.174084073 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/263ef12e-19a9-47c9-adc5-e8b0ec511533-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xrlw2" (UID: "263ef12e-19a9-47c9-adc5-e8b0ec511533") : configmap "kube-rbac-proxy" not found Jul 22 09:18:27.710874 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:27.710556 2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 09:18:27.710874 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:27.710599 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44d91ea8-9844-4ae9-9af8-b5297d3e1639-metrics-tls podName:44d91ea8-9844-4ae9-9af8-b5297d3e1639 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:29.710586042 +0000 UTC m=+40.174380460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44d91ea8-9844-4ae9-9af8-b5297d3e1639-metrics-tls") pod "dns-default-k4mr8" (UID: "44d91ea8-9844-4ae9-9af8-b5297d3e1639") : secret "dns-default-metrics-tls" not found Jul 22 09:18:27.710874 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:27.710659 2549 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 09:18:27.710874 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:27.710670 2549 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d744c845d-jlz5w: secret "image-registry-tls" not found Jul 22 09:18:27.710874 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:27.710698 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-tls podName:5ae4a46e-063d-495a-87c2-f571c93e4850 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:29.7106899 +0000 UTC m=+40.174484331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-tls") pod "image-registry-5d744c845d-jlz5w" (UID: "5ae4a46e-063d-495a-87c2-f571c93e4850") : secret "image-registry-tls" not found Jul 22 09:18:27.710874 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:27.710752 2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 09:18:27.710874 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:27.710776 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/263ef12e-19a9-47c9-adc5-e8b0ec511533-insights-runtime-extractor-tls podName:263ef12e-19a9-47c9-adc5-e8b0ec511533 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:29.710768417 +0000 UTC m=+40.174562832 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/263ef12e-19a9-47c9-adc5-e8b0ec511533-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xrlw2" (UID: "263ef12e-19a9-47c9-adc5-e8b0ec511533") : secret "insights-runtime-extractor-tls" not found Jul 22 09:18:27.743712 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:27.743681 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-5982h"] Jul 22 09:18:27.752991 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:27.752968 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wzwlh"] Jul 22 09:18:27.817181 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:18:27.817152 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a238da_30ed_4a74_9f90_2a32f8076615.slice/crio-03441e8fe8d218cc8e2ed8b9af8b2360610333bf5eb83501a08255c72758fe11 WatchSource:0}: Error finding container 03441e8fe8d218cc8e2ed8b9af8b2360610333bf5eb83501a08255c72758fe11: Status 404 returned error can't find the container with id 03441e8fe8d218cc8e2ed8b9af8b2360610333bf5eb83501a08255c72758fe11 Jul 22 09:18:27.817814 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:18:27.817729 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40f3b1a_129d_42b5_8b2d_99e272295571.slice/crio-c16371be33f5fcb7979748bfb2bcc33f2d4fd354e302e3cc3d844db36ff1d519 WatchSource:0}: Error finding container c16371be33f5fcb7979748bfb2bcc33f2d4fd354e302e3cc3d844db36ff1d519: Status 404 returned error can't find the container with id c16371be33f5fcb7979748bfb2bcc33f2d4fd354e302e3cc3d844db36ff1d519 Jul 22 09:18:28.110979 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:28.110942 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:28.204647 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:28.204562 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:28.379536 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:28.379501 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:28.481287 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:28.481203 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:28.516829 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:28.516791 2549 generic.go:356] "Generic (PLEG): container finished" podID="f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6" containerID="747ee8255f01113e5cb1067249045da5616b79c14772c42fbf585f2481cff425" exitCode=0 Jul 22 09:18:28.516985 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:28.516875 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5f2z" event={"ID":"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6","Type":"ContainerDied","Data":"747ee8255f01113e5cb1067249045da5616b79c14772c42fbf585f2481cff425"} Jul 22 09:18:28.518490 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:28.518225 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-5982h" event={"ID":"19a238da-30ed-4a74-9f90-2a32f8076615","Type":"ContainerStarted","Data":"03441e8fe8d218cc8e2ed8b9af8b2360610333bf5eb83501a08255c72758fe11"} Jul 22 09:18:28.519522 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:28.519497 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wzwlh" event={"ID":"f40f3b1a-129d-42b5-8b2d-99e272295571","Type":"ContainerStarted","Data":"c16371be33f5fcb7979748bfb2bcc33f2d4fd354e302e3cc3d844db36ff1d519"} Jul 22 09:18:28.582480 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:28.582446 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:28.693432 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:28.693399 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:28.806993 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:28.806954 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:29.525514 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:29.525433 2549 generic.go:356] "Generic (PLEG): container finished" podID="f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6" containerID="fec969fb33442c7a8320a5f11edc0cfd341b6ae8d409f7107b1dc8e165478544" exitCode=0 Jul 22 09:18:29.525514 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:29.525479 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5f2z" event={"ID":"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6","Type":"ContainerDied","Data":"fec969fb33442c7a8320a5f11edc0cfd341b6ae8d409f7107b1dc8e165478544"} Jul 22 09:18:29.726255 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:29.726213 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44d91ea8-9844-4ae9-9af8-b5297d3e1639-metrics-tls\") pod \"dns-default-k4mr8\" (UID: \"44d91ea8-9844-4ae9-9af8-b5297d3e1639\") " pod="openshift-dns/dns-default-k4mr8" Jul 22 09:18:29.726441 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:29.726262 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-tls\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:29.726441 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:29.726327 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/263ef12e-19a9-47c9-adc5-e8b0ec511533-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:29.726441 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:29.726385 2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 09:18:29.726441 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:29.726414 2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 09:18:29.726441 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:29.726435 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/263ef12e-19a9-47c9-adc5-e8b0ec511533-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:29.726656 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:29.726417 2549 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 09:18:29.726656 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:29.726461 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44d91ea8-9844-4ae9-9af8-b5297d3e1639-metrics-tls podName:44d91ea8-9844-4ae9-9af8-b5297d3e1639 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:33.726439982 +0000 UTC m=+44.190234402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44d91ea8-9844-4ae9-9af8-b5297d3e1639-metrics-tls") pod "dns-default-k4mr8" (UID: "44d91ea8-9844-4ae9-9af8-b5297d3e1639") : secret "dns-default-metrics-tls" not found Jul 22 09:18:29.726656 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:29.726466 2549 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d744c845d-jlz5w: secret "image-registry-tls" not found Jul 22 09:18:29.726656 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:29.726497 2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 09:18:29.726656 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:29.726510 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/263ef12e-19a9-47c9-adc5-e8b0ec511533-insights-runtime-extractor-tls podName:263ef12e-19a9-47c9-adc5-e8b0ec511533 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:33.726494621 +0000 UTC m=+44.190289041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/263ef12e-19a9-47c9-adc5-e8b0ec511533-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xrlw2" (UID: "263ef12e-19a9-47c9-adc5-e8b0ec511533") : secret "insights-runtime-extractor-tls" not found Jul 22 09:18:29.726656 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:29.726534 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-tls podName:5ae4a46e-063d-495a-87c2-f571c93e4850 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:33.726524077 +0000 UTC m=+44.190318497 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-tls") pod "image-registry-5d744c845d-jlz5w" (UID: "5ae4a46e-063d-495a-87c2-f571c93e4850") : secret "image-registry-tls" not found Jul 22 09:18:29.726656 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:29.726550 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/263ef12e-19a9-47c9-adc5-e8b0ec511533-kube-rbac-proxy-cm podName:263ef12e-19a9-47c9-adc5-e8b0ec511533 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:33.726541814 +0000 UTC m=+44.190336230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/263ef12e-19a9-47c9-adc5-e8b0ec511533-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xrlw2" (UID: "263ef12e-19a9-47c9-adc5-e8b0ec511533") : configmap "kube-rbac-proxy" not found Jul 22 09:18:29.802903 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:29.802866 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:30.198397 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:30.198363 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:30.405967 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:30.405924 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:30.599145 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:30.599102 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:30.837330 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:30.837281 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:30.953698 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:30.953668 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:31.054192 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:31.054161 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:31.409449 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:31.409413 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:31.531987 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:31.531950 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wzwlh" event={"ID":"f40f3b1a-129d-42b5-8b2d-99e272295571","Type":"ContainerStarted","Data":"df84152fcfae08c3aa19adba720ebcb711d305c2419b2401a6177a555e2d24f8"} Jul 22 09:18:31.535237 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:31.535208 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b5f2z" event={"ID":"f69d8767-1133-4d2d-9ebd-5fe2a5fb6ec6","Type":"ContainerStarted","Data":"42c9d4944b1354e25fd3dc235603e5560e8fa188649894b65a392f0cdb6d26b1"} Jul 22 09:18:31.536877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:31.536850 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-5982h" event={"ID":"19a238da-30ed-4a74-9f90-2a32f8076615","Type":"ContainerStarted","Data":"d5a1c23795e08ca128369c32f84cb4762f76fb1933589f1892ea5d06eb1aa082"} Jul 22 09:18:31.536969 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:31.536900 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-5982h" event={"ID":"19a238da-30ed-4a74-9f90-2a32f8076615","Type":"ContainerStarted","Data":"7e100c1aaa5d6954238d23fcef97b89e4dd41961036b5ba207bd9604b2a717b8"} Jul 22 09:18:31.550983 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:31.550937 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wzwlh" podStartSLOduration=3.704940025 podStartE2EDuration="6.550924036s" totalImagesPullingTime="2.845984011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:18:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:31.550702952 +0000 UTC m=+42.014497390" watchObservedRunningTime="2026-07-22 09:18:31.550924036 +0000 UTC m=+42.014718473" Jul 22 09:18:31.570449 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:31.570396 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-5982h" podStartSLOduration=2.731649989 podStartE2EDuration="5.570379793s" totalImagesPullingTime="2.838729804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:31.569924295 +0000 UTC m=+42.033718734" watchObservedRunningTime="2026-07-22 09:18:31.570379793 +0000 UTC m=+42.034174227" Jul 22 09:18:31.595430 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:31.595111 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b5f2z" podStartSLOduration=9.334254943 podStartE2EDuration="41.595090312s" totalImagesPullingTime="32.260835369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:17:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:31.593555279 +0000 UTC m=+42.057349716" watchObservedRunningTime="2026-07-22 09:18:31.595090312 +0000 UTC m=+42.058884753" Jul 22 09:18:31.730838 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:31.730753 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:31.830836 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:31.830794 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:31.997553 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:31.997462 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:32.283905 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:32.283863 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:32.414956 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:32.414918 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:32.516161 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:32.516129 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:32.801378 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:32.801341 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:32.993651 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:32.993602 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:33.186982 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:33.186895 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:33.761438 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:33.761399 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/263ef12e-19a9-47c9-adc5-e8b0ec511533-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:33.761626 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:33.761484 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44d91ea8-9844-4ae9-9af8-b5297d3e1639-metrics-tls\") pod \"dns-default-k4mr8\" (UID: \"44d91ea8-9844-4ae9-9af8-b5297d3e1639\") " pod="openshift-dns/dns-default-k4mr8" Jul 22 09:18:33.761626 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:33.761513 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-tls\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:33.761626 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:33.761524 2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 09:18:33.761626 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:33.761555 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/263ef12e-19a9-47c9-adc5-e8b0ec511533-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:33.761626 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:33.761592 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/263ef12e-19a9-47c9-adc5-e8b0ec511533-kube-rbac-proxy-cm podName:263ef12e-19a9-47c9-adc5-e8b0ec511533 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:41.761572333 +0000 UTC m=+52.225366759 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/263ef12e-19a9-47c9-adc5-e8b0ec511533-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xrlw2" (UID: "263ef12e-19a9-47c9-adc5-e8b0ec511533") : configmap "kube-rbac-proxy" not found Jul 22 09:18:33.761792 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:33.761693 2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 09:18:33.761792 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:18:33.761762 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/263ef12e-19a9-47c9-adc5-e8b0ec511533-insights-runtime-extractor-tls podName:263ef12e-19a9-47c9-adc5-e8b0ec511533 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:41.761743902 +0000 UTC m=+52.225538337 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/263ef12e-19a9-47c9-adc5-e8b0ec511533-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xrlw2" (UID: "263ef12e-19a9-47c9-adc5-e8b0ec511533") : secret "insights-runtime-extractor-tls" not found Jul 22 09:18:33.765322 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:33.765280 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44d91ea8-9844-4ae9-9af8-b5297d3e1639-metrics-tls\") pod \"dns-default-k4mr8\" (UID: \"44d91ea8-9844-4ae9-9af8-b5297d3e1639\") " pod="openshift-dns/dns-default-k4mr8" Jul 22 09:18:33.765457 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:33.765436 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-tls\") pod \"image-registry-5d744c845d-jlz5w\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:33.770851 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:33.770830 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k4mr8" Jul 22 09:18:33.925633 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:33.925603 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k4mr8"] Jul 22 09:18:33.928477 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:18:33.928444 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d91ea8_9844_4ae9_9af8_b5297d3e1639.slice/crio-bc62d26ddee7223d8d65ff1e5d80edbd9b8c94e6334b62b273b0fd4b8eb5c81b WatchSource:0}: Error finding container bc62d26ddee7223d8d65ff1e5d80edbd9b8c94e6334b62b273b0fd4b8eb5c81b: Status 404 returned error can't find the container with id bc62d26ddee7223d8d65ff1e5d80edbd9b8c94e6334b62b273b0fd4b8eb5c81b Jul 22 09:18:34.035763 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:34.035730 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:34.204459 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:34.204424 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:34.211534 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:34.211509 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d744c845d-jlz5w"] Jul 22 09:18:34.214337 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:18:34.214314 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae4a46e_063d_495a_87c2_f571c93e4850.slice/crio-22e7b5d0fc343a7e1490629d9d961ef050d99544a8567ca470355833fa54ab36 WatchSource:0}: Error finding container 22e7b5d0fc343a7e1490629d9d961ef050d99544a8567ca470355833fa54ab36: Status 404 returned error can't find the container with id 22e7b5d0fc343a7e1490629d9d961ef050d99544a8567ca470355833fa54ab36 Jul 22 09:18:34.388059 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:34.388024 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:34.450640 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:34.450603 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:34.545890 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:34.545853 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k4mr8" event={"ID":"44d91ea8-9844-4ae9-9af8-b5297d3e1639","Type":"ContainerStarted","Data":"bc62d26ddee7223d8d65ff1e5d80edbd9b8c94e6334b62b273b0fd4b8eb5c81b"} Jul 22 09:18:34.547546 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:34.547516 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" event={"ID":"5ae4a46e-063d-495a-87c2-f571c93e4850","Type":"ContainerStarted","Data":"ab25a1d4d5111e23fea17619911beab4058c650ee256517d8588bfa9dd459ca2"} Jul 22 09:18:34.547689 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:34.547570 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" event={"ID":"5ae4a46e-063d-495a-87c2-f571c93e4850","Type":"ContainerStarted","Data":"22e7b5d0fc343a7e1490629d9d961ef050d99544a8567ca470355833fa54ab36"} Jul 22 09:18:34.547689 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:34.547681 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:34.568674 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:34.568621 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" podStartSLOduration=16.568604309 podStartE2EDuration="16.568604309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:34.567552602 +0000 UTC m=+45.031347042" watchObservedRunningTime="2026-07-22 09:18:34.568604309 +0000 UTC m=+45.032398745" Jul 22 09:18:34.591742 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:34.591649 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:34.794495 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:34.794440 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:35.604110 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:35.604070 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:35.689340 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:35.689278 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:35.990523 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:35.990449 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:36.219298 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:36.219265 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:36.555578 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:36.555543 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k4mr8" event={"ID":"44d91ea8-9844-4ae9-9af8-b5297d3e1639","Type":"ContainerStarted","Data":"97b24e1fbb0709ddcd3de9a9d6a3d32a0d890a41f082bc3369ce960845e09cc5"} Jul 22 09:18:36.555760 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:36.555599 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k4mr8" event={"ID":"44d91ea8-9844-4ae9-9af8-b5297d3e1639","Type":"ContainerStarted","Data":"1ddb347f6a55cdd93eab6d12001d4a43c69ed1e3aa6d6c4270c8d7b262130b9c"} Jul 22 09:18:36.555760 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:36.555689 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-k4mr8" Jul 22 09:18:36.573592 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:36.573544 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-k4mr8" podStartSLOduration=9.319923436 podStartE2EDuration="11.573530159s" totalImagesPullingTime="2.253606723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:18:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:36.572605357 +0000 UTC m=+47.036399795" watchObservedRunningTime="2026-07-22 09:18:36.573530159 +0000 UTC m=+47.037324597" Jul 22 09:18:36.593044 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:36.593010 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:37.114109 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:37.114071 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:37.397873 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:37.397792 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:37.595820 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:37.595781 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:37.797415 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:37.797384 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:37.995106 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:37.995066 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:38.391829 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:38.391797 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:38.488836 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:38.488803 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:38.796641 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:38.796585 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:41.825861 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:41.825819 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/263ef12e-19a9-47c9-adc5-e8b0ec511533-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:41.826248 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:41.825880 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/263ef12e-19a9-47c9-adc5-e8b0ec511533-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:41.826415 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:41.826397 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/263ef12e-19a9-47c9-adc5-e8b0ec511533-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:41.828058 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:41.828037 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/263ef12e-19a9-47c9-adc5-e8b0ec511533-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xrlw2\" (UID: \"263ef12e-19a9-47c9-adc5-e8b0ec511533\") " pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:41.856572 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:41.856549 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xrlw2" Jul 22 09:18:41.989630 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:41.989598 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xrlw2"] Jul 22 09:18:41.993234 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:18:41.993205 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263ef12e_19a9_47c9_adc5_e8b0ec511533.slice/crio-f2406fc8a821d4304400e477567da1034377f655c503a72334790bcd4f896b04 WatchSource:0}: Error finding container f2406fc8a821d4304400e477567da1034377f655c503a72334790bcd4f896b04: Status 404 returned error can't find the container with id f2406fc8a821d4304400e477567da1034377f655c503a72334790bcd4f896b04 Jul 22 09:18:42.578170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:42.578127 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xrlw2" event={"ID":"263ef12e-19a9-47c9-adc5-e8b0ec511533","Type":"ContainerStarted","Data":"f2406fc8a821d4304400e477567da1034377f655c503a72334790bcd4f896b04"} Jul 22 09:18:43.983485 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:43.983453 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:44.113567 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:44.113526 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:44.297688 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:44.297650 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:44.480952 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:44.480903 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:44.661788 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:44.661701 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:44.734702 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:44.734666 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:44.867029 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:44.866991 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:44.967644 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:44.967561 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:45.264250 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.264164 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:45.363945 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.363905 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:45.464191 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.464141 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:45.675074 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.675010 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:45.832667 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.832599 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6644ddbb4b-zlhqw"] Jul 22 09:18:45.858963 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.858931 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-jwgdc"] Jul 22 09:18:45.859179 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.859084 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6644ddbb4b-zlhqw" Jul 22 09:18:45.863589 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.863560 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 22 09:18:45.863779 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.863591 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 22 09:18:45.864237 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.863881 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 22 09:18:45.864237 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.864054 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-656995d64b-4zzwp"] Jul 22 09:18:45.864237 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.864144 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-2gkt7\"" Jul 22 09:18:45.864237 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.864209 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwgdc" Jul 22 09:18:45.865458 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.865259 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 22 09:18:45.867486 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.867403 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6644ddbb4b-zlhqw"] Jul 22 09:18:45.867486 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.867475 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-jwgdc"] Jul 22 09:18:45.867655 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.867500 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-656995d64b-4zzwp"] Jul 22 09:18:45.867655 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.867559 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-656995d64b-4zzwp" Jul 22 09:18:45.868043 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.867833 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 22 09:18:45.868043 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.867990 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-jv2st\"" Jul 22 09:18:45.869969 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.869941 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 22 09:18:45.870767 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.870746 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 22 09:18:45.890609 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.890584 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d744c845d-jlz5w"] Jul 22 09:18:45.934235 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.934157 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:45.936757 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.936726 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-fc8pn"] Jul 22 09:18:45.942938 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.942914 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6"] Jul 22 09:18:45.943069 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.943049 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-fc8pn" Jul 22 09:18:45.945978 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.945959 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 09:18:45.946087 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.945986 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-brmw6\"" Jul 22 09:18:45.946087 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.945962 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 09:18:45.947440 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.947420 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:45.954124 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.954073 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-fc8pn"] Jul 22 09:18:45.957828 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.957783 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6"] Jul 22 09:18:45.962708 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.962683 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/adc69b3e-14e2-4d1d-9a65-7a89d88190ff-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-jwgdc\" (UID: \"adc69b3e-14e2-4d1d-9a65-7a89d88190ff\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwgdc" Jul 22 09:18:45.962832 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.962744 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5682593f-0351-424b-a6f5-feb96af79ddb-klusterlet-config\") pod \"klusterlet-addon-workmgr-656995d64b-4zzwp\" (UID: \"5682593f-0351-424b-a6f5-feb96af79ddb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-656995d64b-4zzwp" Jul 22 09:18:45.962832 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.962789 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk99x\" (UniqueName: \"kubernetes.io/projected/5682593f-0351-424b-a6f5-feb96af79ddb-kube-api-access-hk99x\") pod \"klusterlet-addon-workmgr-656995d64b-4zzwp\" (UID: \"5682593f-0351-424b-a6f5-feb96af79ddb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-656995d64b-4zzwp" Jul 22 09:18:45.962948 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.962828 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/adc69b3e-14e2-4d1d-9a65-7a89d88190ff-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-jwgdc\" (UID: \"adc69b3e-14e2-4d1d-9a65-7a89d88190ff\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwgdc" Jul 22 09:18:45.962948 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.962932 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4129b4b2-d2fe-43b0-bf70-321e2992ce5d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6644ddbb4b-zlhqw\" (UID: \"4129b4b2-d2fe-43b0-bf70-321e2992ce5d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6644ddbb4b-zlhqw" Jul 22 09:18:45.963052 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.962964 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5682593f-0351-424b-a6f5-feb96af79ddb-tmp\") pod \"klusterlet-addon-workmgr-656995d64b-4zzwp\" (UID: \"5682593f-0351-424b-a6f5-feb96af79ddb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-656995d64b-4zzwp" Jul 22 09:18:45.963052 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:45.962990 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdh5d\" (UniqueName: \"kubernetes.io/projected/4129b4b2-d2fe-43b0-bf70-321e2992ce5d-kube-api-access-pdh5d\") pod \"managed-serviceaccount-addon-agent-6644ddbb4b-zlhqw\" (UID: \"4129b4b2-d2fe-43b0-bf70-321e2992ce5d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6644ddbb4b-zlhqw" Jul 22 09:18:46.028857 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.028810 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:46.063925 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.063886 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5682593f-0351-424b-a6f5-feb96af79ddb-tmp\") pod \"klusterlet-addon-workmgr-656995d64b-4zzwp\" (UID: \"5682593f-0351-424b-a6f5-feb96af79ddb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-656995d64b-4zzwp" Jul 22 09:18:46.063925 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.063925 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdh5d\" (UniqueName: \"kubernetes.io/projected/4129b4b2-d2fe-43b0-bf70-321e2992ce5d-kube-api-access-pdh5d\") pod \"managed-serviceaccount-addon-agent-6644ddbb4b-zlhqw\" (UID: \"4129b4b2-d2fe-43b0-bf70-321e2992ce5d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6644ddbb4b-zlhqw" Jul 22 09:18:46.064181 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.063952 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/adc69b3e-14e2-4d1d-9a65-7a89d88190ff-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-jwgdc\" (UID: \"adc69b3e-14e2-4d1d-9a65-7a89d88190ff\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwgdc" Jul 22 09:18:46.064181 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.064061 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28c1f02e-5341-49ff-9bd0-8a188443891e-bound-sa-token\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.064181 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.064108 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5682593f-0351-424b-a6f5-feb96af79ddb-klusterlet-config\") pod \"klusterlet-addon-workmgr-656995d64b-4zzwp\" (UID: \"5682593f-0351-424b-a6f5-feb96af79ddb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-656995d64b-4zzwp" Jul 22 09:18:46.064181 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.064136 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk99x\" (UniqueName: \"kubernetes.io/projected/5682593f-0351-424b-a6f5-feb96af79ddb-kube-api-access-hk99x\") pod \"klusterlet-addon-workmgr-656995d64b-4zzwp\" (UID: \"5682593f-0351-424b-a6f5-feb96af79ddb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-656995d64b-4zzwp" Jul 22 09:18:46.064181 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.064163 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cnsd\" (UniqueName: \"kubernetes.io/projected/3dc0aeb5-0a3f-485f-bf41-6d4c083faee4-kube-api-access-7cnsd\") pod \"downloads-5bc67dcd46-fc8pn\" (UID: \"3dc0aeb5-0a3f-485f-bf41-6d4c083faee4\") " pod="openshift-console/downloads-5bc67dcd46-fc8pn" Jul 22 09:18:46.064467 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.064192 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28c1f02e-5341-49ff-9bd0-8a188443891e-registry-tls\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.064467 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.064215 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28c1f02e-5341-49ff-9bd0-8a188443891e-trusted-ca\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.064467 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.064257 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/adc69b3e-14e2-4d1d-9a65-7a89d88190ff-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-jwgdc\" (UID: \"adc69b3e-14e2-4d1d-9a65-7a89d88190ff\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwgdc" Jul 22 09:18:46.064467 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.064284 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28c1f02e-5341-49ff-9bd0-8a188443891e-installation-pull-secrets\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.064467 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.064417 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5682593f-0351-424b-a6f5-feb96af79ddb-tmp\") pod \"klusterlet-addon-workmgr-656995d64b-4zzwp\" (UID: \"5682593f-0351-424b-a6f5-feb96af79ddb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-656995d64b-4zzwp" Jul 22 09:18:46.064467 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.064454 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnl6l\" (UniqueName: \"kubernetes.io/projected/28c1f02e-5341-49ff-9bd0-8a188443891e-kube-api-access-qnl6l\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.064764 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.064492 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28c1f02e-5341-49ff-9bd0-8a188443891e-registry-certificates\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.064764 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.064530 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28c1f02e-5341-49ff-9bd0-8a188443891e-image-registry-private-configuration\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.064764 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.064643 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4129b4b2-d2fe-43b0-bf70-321e2992ce5d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6644ddbb4b-zlhqw\" (UID: \"4129b4b2-d2fe-43b0-bf70-321e2992ce5d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6644ddbb4b-zlhqw" Jul 22 09:18:46.064764 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.064671 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28c1f02e-5341-49ff-9bd0-8a188443891e-ca-trust-extracted\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.064764 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.064678 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/adc69b3e-14e2-4d1d-9a65-7a89d88190ff-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-jwgdc\" (UID: \"adc69b3e-14e2-4d1d-9a65-7a89d88190ff\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwgdc" Jul 22 09:18:46.067385 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.067360 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5682593f-0351-424b-a6f5-feb96af79ddb-klusterlet-config\") pod \"klusterlet-addon-workmgr-656995d64b-4zzwp\" (UID: \"5682593f-0351-424b-a6f5-feb96af79ddb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-656995d64b-4zzwp" Jul 22 09:18:46.067527 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.067512 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/adc69b3e-14e2-4d1d-9a65-7a89d88190ff-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-jwgdc\" (UID: \"adc69b3e-14e2-4d1d-9a65-7a89d88190ff\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwgdc" Jul 22 09:18:46.067796 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.067773 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4129b4b2-d2fe-43b0-bf70-321e2992ce5d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6644ddbb4b-zlhqw\" (UID: \"4129b4b2-d2fe-43b0-bf70-321e2992ce5d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6644ddbb4b-zlhqw" Jul 22 09:18:46.075133 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.075112 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdh5d\" (UniqueName: \"kubernetes.io/projected/4129b4b2-d2fe-43b0-bf70-321e2992ce5d-kube-api-access-pdh5d\") pod \"managed-serviceaccount-addon-agent-6644ddbb4b-zlhqw\" (UID: \"4129b4b2-d2fe-43b0-bf70-321e2992ce5d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6644ddbb4b-zlhqw" Jul 22 09:18:46.075267 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.075173 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk99x\" (UniqueName: \"kubernetes.io/projected/5682593f-0351-424b-a6f5-feb96af79ddb-kube-api-access-hk99x\") pod \"klusterlet-addon-workmgr-656995d64b-4zzwp\" (UID: \"5682593f-0351-424b-a6f5-feb96af79ddb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-656995d64b-4zzwp" Jul 22 09:18:46.126359 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.126290 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:46.167672 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.167633 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnl6l\" (UniqueName: \"kubernetes.io/projected/28c1f02e-5341-49ff-9bd0-8a188443891e-kube-api-access-qnl6l\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.167879 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.167689 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28c1f02e-5341-49ff-9bd0-8a188443891e-registry-certificates\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.167879 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.167715 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28c1f02e-5341-49ff-9bd0-8a188443891e-image-registry-private-configuration\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.167879 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.167785 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28c1f02e-5341-49ff-9bd0-8a188443891e-ca-trust-extracted\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.167879 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.167832 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28c1f02e-5341-49ff-9bd0-8a188443891e-bound-sa-token\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.167879 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.167869 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cnsd\" (UniqueName: \"kubernetes.io/projected/3dc0aeb5-0a3f-485f-bf41-6d4c083faee4-kube-api-access-7cnsd\") pod \"downloads-5bc67dcd46-fc8pn\" (UID: \"3dc0aeb5-0a3f-485f-bf41-6d4c083faee4\") " pod="openshift-console/downloads-5bc67dcd46-fc8pn" Jul 22 09:18:46.168140 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.167895 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28c1f02e-5341-49ff-9bd0-8a188443891e-registry-tls\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.168140 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.167915 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28c1f02e-5341-49ff-9bd0-8a188443891e-trusted-ca\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.168140 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.167955 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28c1f02e-5341-49ff-9bd0-8a188443891e-installation-pull-secrets\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.168850 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.168549 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28c1f02e-5341-49ff-9bd0-8a188443891e-ca-trust-extracted\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.168850 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.168759 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28c1f02e-5341-49ff-9bd0-8a188443891e-registry-certificates\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.169233 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.169211 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28c1f02e-5341-49ff-9bd0-8a188443891e-trusted-ca\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.170650 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.170624 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28c1f02e-5341-49ff-9bd0-8a188443891e-image-registry-private-configuration\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.170994 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.170970 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28c1f02e-5341-49ff-9bd0-8a188443891e-installation-pull-secrets\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.171850 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.171829 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28c1f02e-5341-49ff-9bd0-8a188443891e-registry-tls\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.179243 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.179196 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6644ddbb4b-zlhqw" Jul 22 09:18:46.179948 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.179926 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28c1f02e-5341-49ff-9bd0-8a188443891e-bound-sa-token\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.180144 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.180126 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnl6l\" (UniqueName: \"kubernetes.io/projected/28c1f02e-5341-49ff-9bd0-8a188443891e-kube-api-access-qnl6l\") pod \"image-registry-5dc8c5f9bf-7rhc6\" (UID: \"28c1f02e-5341-49ff-9bd0-8a188443891e\") " pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.180470 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.180451 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cnsd\" (UniqueName: \"kubernetes.io/projected/3dc0aeb5-0a3f-485f-bf41-6d4c083faee4-kube-api-access-7cnsd\") pod \"downloads-5bc67dcd46-fc8pn\" (UID: \"3dc0aeb5-0a3f-485f-bf41-6d4c083faee4\") " pod="openshift-console/downloads-5bc67dcd46-fc8pn" Jul 22 09:18:46.187058 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.187009 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwgdc" Jul 22 09:18:46.193937 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.193919 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-656995d64b-4zzwp" Jul 22 09:18:46.255632 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.255596 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-fc8pn" Jul 22 09:18:46.262822 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.262797 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:46.538520 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.538477 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:46.561908 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.561869 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-k4mr8" Jul 22 09:18:46.638153 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.638122 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:46.704833 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:46.704802 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:47.181063 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:47.181031 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-jwgdc"] Jul 22 09:18:47.185164 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:18:47.185136 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc69b3e_14e2_4d1d_9a65_7a89d88190ff.slice/crio-c1e772484383ead6be495ad29ab10ca8857db92ee953ad0b0ab46ed0cc556468 WatchSource:0}: Error finding container c1e772484383ead6be495ad29ab10ca8857db92ee953ad0b0ab46ed0cc556468: Status 404 returned error can't find the container with id c1e772484383ead6be495ad29ab10ca8857db92ee953ad0b0ab46ed0cc556468 Jul 22 09:18:47.187680 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:47.187656 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6644ddbb4b-zlhqw"] Jul 22 09:18:47.192799 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:18:47.192493 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4129b4b2_d2fe_43b0_bf70_321e2992ce5d.slice/crio-58a966f30fc7a5e985997ee1025155ea4a7cb72ad2e37a1ed53649784e51213c WatchSource:0}: Error finding container 58a966f30fc7a5e985997ee1025155ea4a7cb72ad2e37a1ed53649784e51213c: Status 404 returned error can't find the container with id 58a966f30fc7a5e985997ee1025155ea4a7cb72ad2e37a1ed53649784e51213c Jul 22 09:18:47.233017 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:47.232918 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-656995d64b-4zzwp"] Jul 22 09:18:47.235349 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:18:47.235320 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5682593f_0351_424b_a6f5_feb96af79ddb.slice/crio-0ee263485f869026e6a913b106bdc2a7b2c8ee6744cf3b2d2e1c28933b6b735b WatchSource:0}: Error finding container 0ee263485f869026e6a913b106bdc2a7b2c8ee6744cf3b2d2e1c28933b6b735b: Status 404 returned error can't find the container with id 0ee263485f869026e6a913b106bdc2a7b2c8ee6744cf3b2d2e1c28933b6b735b Jul 22 09:18:47.403371 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:47.403343 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-fc8pn"] Jul 22 09:18:47.406233 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:18:47.406185 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc0aeb5_0a3f_485f_bf41_6d4c083faee4.slice/crio-b5d1daaf8223f92ba71fe2b3adb3e3498d775f8a5d90240c2b5c012d91580a60 WatchSource:0}: Error finding container b5d1daaf8223f92ba71fe2b3adb3e3498d775f8a5d90240c2b5c012d91580a60: Status 404 returned error can't find the container with id b5d1daaf8223f92ba71fe2b3adb3e3498d775f8a5d90240c2b5c012d91580a60 Jul 22 09:18:47.407657 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:47.407634 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6"] Jul 22 09:18:47.409953 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:18:47.409932 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c1f02e_5341_49ff_9bd0_8a188443891e.slice/crio-f479c58f508b34345901a84a61680f7d4d08847640b13f58540a5f73bf7a6a20 WatchSource:0}: Error finding container f479c58f508b34345901a84a61680f7d4d08847640b13f58540a5f73bf7a6a20: Status 404 returned error can't find the container with id f479c58f508b34345901a84a61680f7d4d08847640b13f58540a5f73bf7a6a20 Jul 22 09:18:47.602169 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:47.602125 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" event={"ID":"28c1f02e-5341-49ff-9bd0-8a188443891e","Type":"ContainerStarted","Data":"b54406ab71672410f0226f39473d0f833ee35988afc2675cfc5793f01105de2f"} Jul 22 09:18:47.602628 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:47.602194 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" event={"ID":"28c1f02e-5341-49ff-9bd0-8a188443891e","Type":"ContainerStarted","Data":"f479c58f508b34345901a84a61680f7d4d08847640b13f58540a5f73bf7a6a20"} Jul 22 09:18:47.602902 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:47.602876 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:18:47.604551 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:47.604493 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-fc8pn" event={"ID":"3dc0aeb5-0a3f-485f-bf41-6d4c083faee4","Type":"ContainerStarted","Data":"b5d1daaf8223f92ba71fe2b3adb3e3498d775f8a5d90240c2b5c012d91580a60"} Jul 22 09:18:47.606250 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:47.606225 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xrlw2" event={"ID":"263ef12e-19a9-47c9-adc5-e8b0ec511533","Type":"ContainerStarted","Data":"effe6f7d52278cb1d249a94eaea2c81a781420ea1a1ccec1eedcc53831888367"} Jul 22 09:18:47.607787 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:47.607754 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwgdc" event={"ID":"adc69b3e-14e2-4d1d-9a65-7a89d88190ff","Type":"ContainerStarted","Data":"c1e772484383ead6be495ad29ab10ca8857db92ee953ad0b0ab46ed0cc556468"} Jul 22 09:18:47.610806 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:47.610754 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-656995d64b-4zzwp" event={"ID":"5682593f-0351-424b-a6f5-feb96af79ddb","Type":"ContainerStarted","Data":"0ee263485f869026e6a913b106bdc2a7b2c8ee6744cf3b2d2e1c28933b6b735b"} Jul 22 09:18:47.612082 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:47.612041 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6644ddbb4b-zlhqw" event={"ID":"4129b4b2-d2fe-43b0-bf70-321e2992ce5d","Type":"ContainerStarted","Data":"58a966f30fc7a5e985997ee1025155ea4a7cb72ad2e37a1ed53649784e51213c"} Jul 22 09:18:47.628855 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:47.628235 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" podStartSLOduration=2.628213425 podStartE2EDuration="2.628213425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:47.62654853 +0000 UTC m=+58.090342971" watchObservedRunningTime="2026-07-22 09:18:47.628213425 +0000 UTC m=+58.092007848" Jul 22 09:18:48.697767 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:48.697599 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:48.762830 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:48.762646 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:49.629629 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:49.629563 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwgdc" event={"ID":"adc69b3e-14e2-4d1d-9a65-7a89d88190ff","Type":"ContainerStarted","Data":"25e586a6857956f83641f258aa5982a37f3a6c2c99a0bd3a08c3a605090f70e2"} Jul 22 09:18:49.648242 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:49.647316 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwgdc" podStartSLOduration=2.693706793 podStartE2EDuration="4.64727773s" totalImagesPullingTime="1.953570937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:49.646524587 +0000 UTC m=+60.110319026" watchObservedRunningTime="2026-07-22 09:18:49.64727773 +0000 UTC m=+60.111072169" Jul 22 09:18:49.783732 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:49.783683 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:49.853994 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:49.853909 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:50.887772 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:50.887738 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:50.988953 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:50.988921 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:51.510975 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:51.510947 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bcsgw" Jul 22 09:18:51.687969 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:51.687916 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:51.785116 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:51.785082 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:54.091357 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:54.091295 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:55.326124 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.326078 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:55.485768 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.485729 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:55.659486 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.659404 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xrlw2" event={"ID":"263ef12e-19a9-47c9-adc5-e8b0ec511533","Type":"ContainerStarted","Data":"3e50d0209d7829127916d868d60282b8c9ceaa6f8c98fa75cdb54e173d69f9f3"} Jul 22 09:18:55.661371 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.661344 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-656995d64b-4zzwp" event={"ID":"5682593f-0351-424b-a6f5-feb96af79ddb","Type":"ContainerStarted","Data":"01c7acf5dae3997475b9dfff2843d447cb005af5a37b3227c8b6279a04f8a891"} Jul 22 09:18:55.661650 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.661608 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-656995d64b-4zzwp" Jul 22 09:18:55.663677 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.663651 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6644ddbb4b-zlhqw" event={"ID":"4129b4b2-d2fe-43b0-bf70-321e2992ce5d","Type":"ContainerStarted","Data":"8c43caebbf095a3c444b4fb8a7a038f71519d597c474220f0487b9a02f66243d"} Jul 22 09:18:55.663793 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.663758 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-656995d64b-4zzwp" Jul 22 09:18:55.681533 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.681478 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-656995d64b-4zzwp" podStartSLOduration=3.22705442 podStartE2EDuration="10.681459212s" totalImagesPullingTime="7.454404792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:55.679854611 +0000 UTC m=+66.143649050" watchObservedRunningTime="2026-07-22 09:18:55.681459212 +0000 UTC m=+66.145253655" Jul 22 09:18:55.699624 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.699575 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6644ddbb4b-zlhqw" podStartSLOduration=3.203543951 podStartE2EDuration="10.699554988s" totalImagesPullingTime="7.496011037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:55.69904023 +0000 UTC m=+66.162834669" watchObservedRunningTime="2026-07-22 09:18:55.699554988 +0000 UTC m=+66.163349425" Jul 22 09:18:55.708149 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.708091 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:55.770671 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.770628 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret\") pod \"global-pull-secret-syncer-49l4x\" (UID: \"a5680bfb-a669-4640-876e-84a6322b2454\") " pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:18:55.770871 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.770761 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs\") pod \"network-metrics-daemon-6sjqx\" (UID: \"54cff721-5d48-4e7a-9771-1b859bc68609\") " pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:18:55.773664 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.773477 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 09:18:55.773664 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.773474 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 09:18:55.783631 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.783576 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5680bfb-a669-4640-876e-84a6322b2454-original-pull-secret\") pod \"global-pull-secret-syncer-49l4x\" (UID: \"a5680bfb-a669-4640-876e-84a6322b2454\") " pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:18:55.784200 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.784136 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54cff721-5d48-4e7a-9771-1b859bc68609-metrics-certs\") pod \"network-metrics-daemon-6sjqx\" (UID: \"54cff721-5d48-4e7a-9771-1b859bc68609\") " pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:18:55.805924 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.805886 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:55.847634 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.847601 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6sgn5\"" Jul 22 09:18:55.855908 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.855878 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6sjqx" Jul 22 09:18:55.858199 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.858152 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49l4x" Jul 22 09:18:55.871970 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.871516 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvhx\" (UniqueName: \"kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx\") pod \"network-check-target-mrw5j\" (UID: \"2e590737-a6b2-482a-9cf2-23d1add97650\") " pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:18:55.874640 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.874619 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 09:18:55.884511 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.884484 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 09:18:55.896357 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.895944 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zvhx\" (UniqueName: \"kubernetes.io/projected/2e590737-a6b2-482a-9cf2-23d1add97650-kube-api-access-5zvhx\") pod \"network-check-target-mrw5j\" (UID: \"2e590737-a6b2-482a-9cf2-23d1add97650\") " pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:18:55.898426 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:55.898396 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:18:56.030091 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:56.030004 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6sjqx"] Jul 22 09:18:56.033924 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:18:56.033894 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54cff721_5d48_4e7a_9771_1b859bc68609.slice/crio-d89fa78fbf8e43ab868c529d5d403c3820b58d88a7d53c8ca158732f156ee6a6 WatchSource:0}: Error finding container d89fa78fbf8e43ab868c529d5d403c3820b58d88a7d53c8ca158732f156ee6a6: Status 404 returned error can't find the container with id d89fa78fbf8e43ab868c529d5d403c3820b58d88a7d53c8ca158732f156ee6a6 Jul 22 09:18:56.051844 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:56.051811 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-49l4x"] Jul 22 09:18:56.055281 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:18:56.055251 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5680bfb_a669_4640_876e_84a6322b2454.slice/crio-478b80b93edaf57a454e1b08a0bd704125173edd462d1692eb959d1faa1f5b9b WatchSource:0}: Error finding container 478b80b93edaf57a454e1b08a0bd704125173edd462d1692eb959d1faa1f5b9b: Status 404 returned error can't find the container with id 478b80b93edaf57a454e1b08a0bd704125173edd462d1692eb959d1faa1f5b9b Jul 22 09:18:56.154502 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:56.154470 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8b4k6\"" Jul 22 09:18:56.161834 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:56.161806 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:18:56.306167 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:56.305982 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:56.316891 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:56.316859 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mrw5j"] Jul 22 09:18:56.320872 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:18:56.320836 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e590737_a6b2_482a_9cf2_23d1add97650.slice/crio-a48941e03c5347936e0f06fa9dbced9f449bc8c2b8bf89d967b4a3184b4293b8 WatchSource:0}: Error finding container a48941e03c5347936e0f06fa9dbced9f449bc8c2b8bf89d967b4a3184b4293b8: Status 404 returned error can't find the container with id a48941e03c5347936e0f06fa9dbced9f449bc8c2b8bf89d967b4a3184b4293b8 Jul 22 09:18:56.405922 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:56.405878 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:56.672766 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:56.672724 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mrw5j" event={"ID":"2e590737-a6b2-482a-9cf2-23d1add97650","Type":"ContainerStarted","Data":"a48941e03c5347936e0f06fa9dbced9f449bc8c2b8bf89d967b4a3184b4293b8"} Jul 22 09:18:56.674350 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:56.674269 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-49l4x" event={"ID":"a5680bfb-a669-4640-876e-84a6322b2454","Type":"ContainerStarted","Data":"478b80b93edaf57a454e1b08a0bd704125173edd462d1692eb959d1faa1f5b9b"} Jul 22 09:18:56.676402 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:56.676375 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6sjqx" event={"ID":"54cff721-5d48-4e7a-9771-1b859bc68609","Type":"ContainerStarted","Data":"d89fa78fbf8e43ab868c529d5d403c3820b58d88a7d53c8ca158732f156ee6a6"} Jul 22 09:18:57.529275 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:57.529237 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:57.637119 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:57.637081 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:57.794314 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:57.794221 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:18:59.900640 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:18:59.900594 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:00.006967 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:00.006922 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:00.141105 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:00.141067 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:00.235480 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:00.235373 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:01.935093 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:01.935054 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:02.036058 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:02.036017 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:02.677492 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:02.677447 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:02.929982 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:02.929901 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-njt9w"] Jul 22 09:19:02.937349 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:02.937325 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:02.940118 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:02.940097 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 09:19:02.940251 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:02.940149 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 09:19:02.940251 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:02.940105 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 09:19:02.941160 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:02.941014 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 09:19:02.941160 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:02.941025 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 09:19:02.941160 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:02.941070 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 09:19:02.941160 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:02.941121 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jgbts\"" Jul 22 09:19:03.033423 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.033385 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/79573ff8-91b8-42b2-aefd-3c699582c1b9-node-exporter-textfile\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.033599 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.033429 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79573ff8-91b8-42b2-aefd-3c699582c1b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.033599 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.033528 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79573ff8-91b8-42b2-aefd-3c699582c1b9-node-exporter-tls\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.033705 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.033599 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8c6j\" (UniqueName: \"kubernetes.io/projected/79573ff8-91b8-42b2-aefd-3c699582c1b9-kube-api-access-j8c6j\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.033705 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.033623 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79573ff8-91b8-42b2-aefd-3c699582c1b9-sys\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.033705 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.033645 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/79573ff8-91b8-42b2-aefd-3c699582c1b9-root\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.033705 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.033668 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/79573ff8-91b8-42b2-aefd-3c699582c1b9-node-exporter-wtmp\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.033890 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.033706 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/79573ff8-91b8-42b2-aefd-3c699582c1b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.033890 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.033731 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79573ff8-91b8-42b2-aefd-3c699582c1b9-metrics-client-ca\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.134589 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.134556 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79573ff8-91b8-42b2-aefd-3c699582c1b9-node-exporter-tls\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.134754 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.134599 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8c6j\" (UniqueName: \"kubernetes.io/projected/79573ff8-91b8-42b2-aefd-3c699582c1b9-kube-api-access-j8c6j\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.134754 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.134625 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79573ff8-91b8-42b2-aefd-3c699582c1b9-sys\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.134754 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.134646 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/79573ff8-91b8-42b2-aefd-3c699582c1b9-root\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.134754 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.134702 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/79573ff8-91b8-42b2-aefd-3c699582c1b9-root\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.134754 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.134741 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/79573ff8-91b8-42b2-aefd-3c699582c1b9-node-exporter-wtmp\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.134946 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.134756 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79573ff8-91b8-42b2-aefd-3c699582c1b9-sys\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.134946 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.134772 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/79573ff8-91b8-42b2-aefd-3c699582c1b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.134946 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.134793 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79573ff8-91b8-42b2-aefd-3c699582c1b9-metrics-client-ca\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.134946 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.134837 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/79573ff8-91b8-42b2-aefd-3c699582c1b9-node-exporter-textfile\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.134946 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.134864 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79573ff8-91b8-42b2-aefd-3c699582c1b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.134946 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.134904 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/79573ff8-91b8-42b2-aefd-3c699582c1b9-node-exporter-wtmp\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.135255 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.135226 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/79573ff8-91b8-42b2-aefd-3c699582c1b9-node-exporter-textfile\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.135514 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.135480 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/79573ff8-91b8-42b2-aefd-3c699582c1b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.135628 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.135550 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79573ff8-91b8-42b2-aefd-3c699582c1b9-metrics-client-ca\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.137275 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.137244 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79573ff8-91b8-42b2-aefd-3c699582c1b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.137385 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.137337 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79573ff8-91b8-42b2-aefd-3c699582c1b9-node-exporter-tls\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.148087 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.148064 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8c6j\" (UniqueName: \"kubernetes.io/projected/79573ff8-91b8-42b2-aefd-3c699582c1b9-kube-api-access-j8c6j\") pod \"node-exporter-njt9w\" (UID: \"79573ff8-91b8-42b2-aefd-3c699582c1b9\") " pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:03.249489 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:03.249410 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-njt9w" Jul 22 09:19:04.190121 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:04.190084 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:04.251948 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:04.251910 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:04.504298 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:04.504211 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:04.605253 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:04.605202 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:05.912208 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:05.912159 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:06.012968 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:06.012928 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:06.393076 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:06.393039 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:06.633140 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:06.633105 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:07.700966 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:07.700930 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:07.798826 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:07.798787 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:08.109626 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:08.109571 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:09.250343 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:09.250285 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:09.411270 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:09.411238 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:09.520100 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:09.520016 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:09.634448 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:09.634422 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5dc8c5f9bf-7rhc6" Jul 22 09:19:09.778140 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:09.778112 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:09.845881 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:09.845857 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:10.177182 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:19:10.176934 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79573ff8_91b8_42b2_aefd_3c699582c1b9.slice/crio-7471e822ef21b1eb5873eb510668869e7dce2b5ed49afa86249680ab7e613a91 WatchSource:0}: Error finding container 7471e822ef21b1eb5873eb510668869e7dce2b5ed49afa86249680ab7e613a91: Status 404 returned error can't find the container with id 7471e822ef21b1eb5873eb510668869e7dce2b5ed49afa86249680ab7e613a91 Jul 22 09:19:10.677150 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:10.677050 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:10.737368 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:10.736122 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-fc8pn" event={"ID":"3dc0aeb5-0a3f-485f-bf41-6d4c083faee4","Type":"ContainerStarted","Data":"367a7b6e4eebc726b9054286e75cb753266a2dfa9214d647bdcf7bfbb39d32be"} Jul 22 09:19:10.737368 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:10.737041 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-fc8pn" Jul 22 09:19:10.743155 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:10.738941 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mrw5j" event={"ID":"2e590737-a6b2-482a-9cf2-23d1add97650","Type":"ContainerStarted","Data":"98ee4dffa5127f955a1e05576580978fc86beed097a50782145c7fa88c1e0131"} Jul 22 09:19:10.743155 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:10.739604 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:19:10.743155 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:10.741377 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-49l4x" event={"ID":"a5680bfb-a669-4640-876e-84a6322b2454","Type":"ContainerStarted","Data":"c4064b9d88fabe11640fc47abc16a367a41a36cdf7cab2d07b04996fd6eb02d3"} Jul 22 09:19:10.744585 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:10.744561 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6sjqx" event={"ID":"54cff721-5d48-4e7a-9771-1b859bc68609","Type":"ContainerStarted","Data":"2f374bbb37efffda2c5a5847c8d9f2eedaca7297ef3f0e9ffdbb93d59fa28cb1"} Jul 22 09:19:10.744749 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:10.744735 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6sjqx" event={"ID":"54cff721-5d48-4e7a-9771-1b859bc68609","Type":"ContainerStarted","Data":"0da926cd859b749ce2e59e0aab0697fa8d5af6633ca48bdc0f93440b1d3ffd50"} Jul 22 09:19:10.748252 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:10.748202 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xrlw2" event={"ID":"263ef12e-19a9-47c9-adc5-e8b0ec511533","Type":"ContainerStarted","Data":"bcf208e688a4140225f677a07da56782cb8d185468b802fdf1070747e32faadc"} Jul 22 09:19:10.750164 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:10.750039 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-njt9w" event={"ID":"79573ff8-91b8-42b2-aefd-3c699582c1b9","Type":"ContainerStarted","Data":"7471e822ef21b1eb5873eb510668869e7dce2b5ed49afa86249680ab7e613a91"} Jul 22 09:19:10.751832 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:10.751811 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-fc8pn" Jul 22 09:19:10.756936 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:10.756888 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-fc8pn" podStartSLOduration=2.989983964 podStartE2EDuration="25.756871527s" totalImagesPullingTime="22.766887563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:19:10.755621902 +0000 UTC m=+81.219416341" watchObservedRunningTime="2026-07-22 09:19:10.756871527 +0000 UTC m=+81.220665962" Jul 22 09:19:10.764009 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:10.763965 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:10.796520 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:10.796282 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-49l4x" podStartSLOduration=66.677738515 podStartE2EDuration="1m20.796261183s" totalImagesPullingTime="14.118522668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:17:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:19:10.79544711 +0000 UTC m=+81.259241549" watchObservedRunningTime="2026-07-22 09:19:10.796261183 +0000 UTC m=+81.260055644" Jul 22 09:19:10.812345 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:10.812214 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mrw5j" podStartSLOduration=66.95900347 podStartE2EDuration="1m20.812193179s" totalImagesPullingTime="13.853189709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:17:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:19:10.810176341 +0000 UTC m=+81.273970780" watchObservedRunningTime="2026-07-22 09:19:10.812193179 +0000 UTC m=+81.275987619" Jul 22 09:19:10.830929 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:10.830828 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xrlw2" podStartSLOduration=17.829369498 podStartE2EDuration="45.830807179s" totalImagesPullingTime="28.001437681s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:18:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:19:10.828780729 +0000 UTC m=+81.292575179" watchObservedRunningTime="2026-07-22 09:19:10.830807179 +0000 UTC m=+81.294601618" Jul 22 09:19:10.846342 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:10.846258 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6sjqx" podStartSLOduration=66.457930725 podStartE2EDuration="1m20.846223208s" totalImagesPullingTime="14.388292483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:17:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:19:10.845881143 +0000 UTC m=+81.309675584" watchObservedRunningTime="2026-07-22 09:19:10.846223208 +0000 UTC m=+81.310017646" Jul 22 09:19:10.913229 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:10.913191 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" podUID="5ae4a46e-063d-495a-87c2-f571c93e4850" containerName="registry" containerID="cri-o://ab25a1d4d5111e23fea17619911beab4058c650ee256517d8588bfa9dd459ca2" gracePeriod=30 Jul 22 09:19:10.944632 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:10.944603 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:11.024141 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.024102 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:11.618980 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.618934 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:19:11.755873 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.755834 2549 generic.go:356] "Generic (PLEG): container finished" podID="5ae4a46e-063d-495a-87c2-f571c93e4850" containerID="ab25a1d4d5111e23fea17619911beab4058c650ee256517d8588bfa9dd459ca2" exitCode=0 Jul 22 09:19:11.756341 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.755890 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" Jul 22 09:19:11.756341 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.755926 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" event={"ID":"5ae4a46e-063d-495a-87c2-f571c93e4850","Type":"ContainerDied","Data":"ab25a1d4d5111e23fea17619911beab4058c650ee256517d8588bfa9dd459ca2"} Jul 22 09:19:11.756341 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.755990 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d744c845d-jlz5w" event={"ID":"5ae4a46e-063d-495a-87c2-f571c93e4850","Type":"ContainerDied","Data":"22e7b5d0fc343a7e1490629d9d961ef050d99544a8567ca470355833fa54ab36"} Jul 22 09:19:11.756341 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.756018 2549 scope.go:117] "RemoveContainer" containerID="ab25a1d4d5111e23fea17619911beab4058c650ee256517d8588bfa9dd459ca2" Jul 22 09:19:11.758364 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.758298 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-njt9w" event={"ID":"79573ff8-91b8-42b2-aefd-3c699582c1b9","Type":"ContainerStarted","Data":"0b0a111aa111869867840ebcb687c024754603ff98d0b63db5a6e51b2af80633"} Jul 22 09:19:11.766240 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.766151 2549 scope.go:117] "RemoveContainer" containerID="ab25a1d4d5111e23fea17619911beab4058c650ee256517d8588bfa9dd459ca2" Jul 22 09:19:11.766550 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:19:11.766516 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab25a1d4d5111e23fea17619911beab4058c650ee256517d8588bfa9dd459ca2\": container with ID starting with ab25a1d4d5111e23fea17619911beab4058c650ee256517d8588bfa9dd459ca2 not found: ID does not exist" containerID="ab25a1d4d5111e23fea17619911beab4058c650ee256517d8588bfa9dd459ca2" Jul 22 09:19:11.766630 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.766555 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab25a1d4d5111e23fea17619911beab4058c650ee256517d8588bfa9dd459ca2"} err="failed to get container status \"ab25a1d4d5111e23fea17619911beab4058c650ee256517d8588bfa9dd459ca2\": rpc error: code = NotFound desc = could not find container \"ab25a1d4d5111e23fea17619911beab4058c650ee256517d8588bfa9dd459ca2\": container with ID starting with ab25a1d4d5111e23fea17619911beab4058c650ee256517d8588bfa9dd459ca2 not found: ID does not exist" Jul 22 09:19:11.805574 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.805537 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ae4a46e-063d-495a-87c2-f571c93e4850-ca-trust-extracted\") pod \"5ae4a46e-063d-495a-87c2-f571c93e4850\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " Jul 22 09:19:11.805776 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.805604 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-tls\") pod \"5ae4a46e-063d-495a-87c2-f571c93e4850\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " Jul 22 09:19:11.805776 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.805639 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-certificates\") pod \"5ae4a46e-063d-495a-87c2-f571c93e4850\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " Jul 22 09:19:11.805776 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.805698 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ae4a46e-063d-495a-87c2-f571c93e4850-installation-pull-secrets\") pod \"5ae4a46e-063d-495a-87c2-f571c93e4850\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " Jul 22 09:19:11.805776 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.805728 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-bound-sa-token\") pod \"5ae4a46e-063d-495a-87c2-f571c93e4850\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " Jul 22 09:19:11.806145 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.806113 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5ae4a46e-063d-495a-87c2-f571c93e4850" (UID: "5ae4a46e-063d-495a-87c2-f571c93e4850"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:19:11.806145 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.806136 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ae4a46e-063d-495a-87c2-f571c93e4850-trusted-ca\") pod \"5ae4a46e-063d-495a-87c2-f571c93e4850\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " Jul 22 09:19:11.806293 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.806169 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ae4a46e-063d-495a-87c2-f571c93e4850-image-registry-private-configuration\") pod \"5ae4a46e-063d-495a-87c2-f571c93e4850\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " Jul 22 09:19:11.806468 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.806396 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnw66\" (UniqueName: \"kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-kube-api-access-jnw66\") pod \"5ae4a46e-063d-495a-87c2-f571c93e4850\" (UID: \"5ae4a46e-063d-495a-87c2-f571c93e4850\") " Jul 22 09:19:11.806947 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.806919 2549 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-certificates\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:19:11.808527 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.808488 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae4a46e-063d-495a-87c2-f571c93e4850-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5ae4a46e-063d-495a-87c2-f571c93e4850" (UID: "5ae4a46e-063d-495a-87c2-f571c93e4850"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:19:11.808860 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.808835 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5ae4a46e-063d-495a-87c2-f571c93e4850" (UID: "5ae4a46e-063d-495a-87c2-f571c93e4850"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:19:11.808999 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.808975 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae4a46e-063d-495a-87c2-f571c93e4850-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5ae4a46e-063d-495a-87c2-f571c93e4850" (UID: "5ae4a46e-063d-495a-87c2-f571c93e4850"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:19:11.809094 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.809072 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5ae4a46e-063d-495a-87c2-f571c93e4850" (UID: "5ae4a46e-063d-495a-87c2-f571c93e4850"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:19:11.810224 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.810194 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-kube-api-access-jnw66" (OuterVolumeSpecName: "kube-api-access-jnw66") pod "5ae4a46e-063d-495a-87c2-f571c93e4850" (UID: "5ae4a46e-063d-495a-87c2-f571c93e4850"). InnerVolumeSpecName "kube-api-access-jnw66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:19:11.810495 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.810459 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae4a46e-063d-495a-87c2-f571c93e4850-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5ae4a46e-063d-495a-87c2-f571c93e4850" (UID: "5ae4a46e-063d-495a-87c2-f571c93e4850"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:19:11.818265 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.818237 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae4a46e-063d-495a-87c2-f571c93e4850-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5ae4a46e-063d-495a-87c2-f571c93e4850" (UID: "5ae4a46e-063d-495a-87c2-f571c93e4850"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:19:11.907331 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.907234 2549 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ae4a46e-063d-495a-87c2-f571c93e4850-ca-trust-extracted\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:19:11.907331 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.907273 2549 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-registry-tls\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:19:11.907331 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.907286 2549 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ae4a46e-063d-495a-87c2-f571c93e4850-installation-pull-secrets\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:19:11.907331 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.907320 2549 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-bound-sa-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:19:11.907331 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.907334 2549 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ae4a46e-063d-495a-87c2-f571c93e4850-trusted-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:19:11.907661 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.907347 2549 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ae4a46e-063d-495a-87c2-f571c93e4850-image-registry-private-configuration\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:19:11.907661 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:11.907359 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnw66\" (UniqueName: \"kubernetes.io/projected/5ae4a46e-063d-495a-87c2-f571c93e4850-kube-api-access-jnw66\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:19:12.084349 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:12.084292 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d744c845d-jlz5w"] Jul 22 09:19:12.092699 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:12.092668 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d744c845d-jlz5w"] Jul 22 09:19:12.340637 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:12.340599 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ae4a46e-063d-495a-87c2-f571c93e4850" path="/var/lib/kubelet/pods/5ae4a46e-063d-495a-87c2-f571c93e4850/volumes" Jul 22 09:19:12.402320 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:12.402269 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:12.764840 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:12.764753 2549 generic.go:356] "Generic (PLEG): container finished" podID="79573ff8-91b8-42b2-aefd-3c699582c1b9" containerID="0b0a111aa111869867840ebcb687c024754603ff98d0b63db5a6e51b2af80633" exitCode=0 Jul 22 09:19:12.764840 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:12.764818 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-njt9w" event={"ID":"79573ff8-91b8-42b2-aefd-3c699582c1b9","Type":"ContainerDied","Data":"0b0a111aa111869867840ebcb687c024754603ff98d0b63db5a6e51b2af80633"} Jul 22 09:19:13.773594 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:13.773556 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-njt9w" event={"ID":"79573ff8-91b8-42b2-aefd-3c699582c1b9","Type":"ContainerStarted","Data":"45e3a9f7ff3cd21fb78d87de1d77814da31b8aa19d0f3125caf59385ec3cfe3e"} Jul 22 09:19:13.774059 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:13.773648 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-njt9w" event={"ID":"79573ff8-91b8-42b2-aefd-3c699582c1b9","Type":"ContainerStarted","Data":"67401d8a5aab264a0b19deedf77d9855bdb3d8725aa2848dc701b2cba56162a3"} Jul 22 09:19:13.795878 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:13.795824 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-njt9w" podStartSLOduration=10.27522112 podStartE2EDuration="11.795808878s" totalImagesPullingTime="1.426355952s" totalInitContainerRuntime="94.231806ms" isStatefulPod=true podCreationTimestamp="2026-07-22 09:19:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:19:13.794455278 +0000 UTC m=+84.258249715" watchObservedRunningTime="2026-07-22 09:19:13.795808878 +0000 UTC m=+84.259603315" Jul 22 09:19:14.568998 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:14.568948 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:14.670882 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:14.670840 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:18.058911 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:18.058873 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:18.159776 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:18.159738 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:18.716427 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:18.716382 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:18.801632 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:18.801593 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:19.640163 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:19.640129 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:19.720543 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:19.720515 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:19.807582 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:19.807537 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:23.204502 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:23.204455 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:23.281534 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:23.281507 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:25.366847 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:25.366804 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:25.468239 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:25.468207 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:25.571731 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:25.571701 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:27.584212 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:27.584170 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:27.656506 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:27.656459 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:33.686706 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:33.686673 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:33.788286 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:33.788248 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:35.962331 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:35.962270 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:36.033988 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:36.033955 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:41.761778 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:41.761747 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mrw5j" Jul 22 09:19:48.717597 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:48.717557 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:19:48.823708 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:19:48.823665 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:20:16.416435 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:20:16.416388 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:20:22.273195 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:20:22.273156 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:20:22.689501 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:20:22.689450 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:20:31.776104 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:20:31.776064 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:20:31.873816 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:20:31.873779 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:20:32.426674 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:20:32.426643 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:20:32.522651 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:20:32.522613 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:20:43.036750 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:20:43.036670 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:20:43.137814 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:20:43.137776 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:20:43.212968 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:20:43.212934 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:21:04.514862 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:21:04.514819 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:21:06.609999 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:21:06.609955 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:21:06.679860 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:21:06.679826 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:21:15.798444 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:21:15.798410 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:21:15.897651 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:21:15.897628 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:21:37.618032 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:21:37.617981 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:21:37.687753 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:21:37.687713 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:21:44.248572 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:21:44.248531 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:21:44.336970 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:21:44.336935 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:21:49.675588 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:21:49.675545 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:21:49.776963 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:21:49.776923 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:21:50.761931 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:21:50.761900 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:21:55.323083 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:21:55.323047 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:21:55.424430 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:21:55.424389 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:21:55.495490 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:21:55.495454 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:22:03.697409 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:03.697363 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:22:03.797944 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:03.797906 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:22:03.898654 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:03.898626 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:22:15.997196 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:15.997111 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:22:16.082737 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:16.082698 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:22:27.103587 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:27.103551 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:22:27.203203 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:27.203169 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:22:28.355609 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:28.355575 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx"] Jul 22 09:22:28.355957 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:28.355808 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ae4a46e-063d-495a-87c2-f571c93e4850" containerName="registry" Jul 22 09:22:28.355957 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:28.355817 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ae4a46e-063d-495a-87c2-f571c93e4850" containerName="registry" Jul 22 09:22:28.355957 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:28.355859 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ae4a46e-063d-495a-87c2-f571c93e4850" containerName="registry" Jul 22 09:22:28.358728 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:28.358711 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx" Jul 22 09:22:28.361334 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:28.361298 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 09:22:28.361426 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:28.361353 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 09:22:28.362213 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:28.362194 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gjwg9\"" Jul 22 09:22:28.369603 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:28.369580 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx"] Jul 22 09:22:28.375803 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:28.375779 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58ffg\" (UniqueName: \"kubernetes.io/projected/1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a-kube-api-access-58ffg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx\" (UID: \"1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx" Jul 22 09:22:28.375911 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:28.375808 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx\" (UID: \"1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx" Jul 22 09:22:28.375911 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:28.375887 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx\" (UID: \"1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx" Jul 22 09:22:28.476877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:28.476847 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx\" (UID: \"1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx" Jul 22 09:22:28.477058 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:28.476888 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58ffg\" (UniqueName: \"kubernetes.io/projected/1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a-kube-api-access-58ffg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx\" (UID: \"1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx" Jul 22 09:22:28.477058 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:28.476907 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx\" (UID: \"1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx" Jul 22 09:22:28.477236 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:28.477220 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx\" (UID: \"1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx" Jul 22 09:22:28.477273 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:28.477254 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx\" (UID: \"1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx" Jul 22 09:22:28.486083 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:28.486050 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58ffg\" (UniqueName: \"kubernetes.io/projected/1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a-kube-api-access-58ffg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx\" (UID: \"1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx" Jul 22 09:22:28.667903 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:28.667812 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx" Jul 22 09:22:28.797803 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:28.797772 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx"] Jul 22 09:22:28.801236 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:22:28.801213 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2e53b2_62bc_4bf0_8cda_ca4f0d2c300a.slice/crio-3c2cdb0b40b7e34a78e15c10ce1b8c3ae7f086411e9b61f46fb9068ffb2b7c6d WatchSource:0}: Error finding container 3c2cdb0b40b7e34a78e15c10ce1b8c3ae7f086411e9b61f46fb9068ffb2b7c6d: Status 404 returned error can't find the container with id 3c2cdb0b40b7e34a78e15c10ce1b8c3ae7f086411e9b61f46fb9068ffb2b7c6d Jul 22 09:22:29.415064 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:29.415026 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx" event={"ID":"1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a","Type":"ContainerStarted","Data":"3c2cdb0b40b7e34a78e15c10ce1b8c3ae7f086411e9b61f46fb9068ffb2b7c6d"} Jul 22 09:22:35.437813 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:35.437771 2549 generic.go:356] "Generic (PLEG): container finished" podID="1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a" containerID="c87daa4d95e73cd2383b2d74ba98276371a63ae5269a17c831671a86398c8495" exitCode=0 Jul 22 09:22:35.438164 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:35.437833 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx" event={"ID":"1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a","Type":"ContainerDied","Data":"c87daa4d95e73cd2383b2d74ba98276371a63ae5269a17c831671a86398c8495"} Jul 22 09:22:38.448787 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:38.448744 2549 generic.go:356] "Generic (PLEG): container finished" podID="1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a" containerID="f0ebbbc98eae83215070a718ac3dcfca59027397e324c253e8e39dc94a902498" exitCode=0 Jul 22 09:22:38.449187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:38.448801 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx" event={"ID":"1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a","Type":"ContainerDied","Data":"f0ebbbc98eae83215070a718ac3dcfca59027397e324c253e8e39dc94a902498"} Jul 22 09:22:45.476168 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:45.476133 2549 generic.go:356] "Generic (PLEG): container finished" podID="1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a" containerID="4351646b78b45365b030a5f9667a3f354a6ce4ad951787e36623d2d7ef9baeb7" exitCode=0 Jul 22 09:22:45.476574 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:45.476205 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx" event={"ID":"1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a","Type":"ContainerDied","Data":"4351646b78b45365b030a5f9667a3f354a6ce4ad951787e36623d2d7ef9baeb7"} Jul 22 09:22:46.597020 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:46.596997 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx" Jul 22 09:22:46.723179 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:46.723146 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a-util\") pod \"1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a\" (UID: \"1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a\") " Jul 22 09:22:46.723179 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:46.723186 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58ffg\" (UniqueName: \"kubernetes.io/projected/1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a-kube-api-access-58ffg\") pod \"1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a\" (UID: \"1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a\") " Jul 22 09:22:46.723418 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:46.723237 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a-bundle\") pod \"1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a\" (UID: \"1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a\") " Jul 22 09:22:46.723903 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:46.723876 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a-bundle" (OuterVolumeSpecName: "bundle") pod "1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a" (UID: "1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:22:46.725491 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:46.725464 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a-kube-api-access-58ffg" (OuterVolumeSpecName: "kube-api-access-58ffg") pod "1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a" (UID: "1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a"). InnerVolumeSpecName "kube-api-access-58ffg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:22:46.727143 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:46.727121 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a-util" (OuterVolumeSpecName: "util") pod "1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a" (UID: "1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:22:46.823958 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:46.823922 2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a-bundle\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:22:46.823958 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:46.823950 2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a-util\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:22:46.823958 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:46.823960 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58ffg\" (UniqueName: \"kubernetes.io/projected/1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a-kube-api-access-58ffg\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:22:47.484358 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:47.484332 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx" Jul 22 09:22:47.484594 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:47.484298 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407f4bx" event={"ID":"1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a","Type":"ContainerDied","Data":"3c2cdb0b40b7e34a78e15c10ce1b8c3ae7f086411e9b61f46fb9068ffb2b7c6d"} Jul 22 09:22:47.484594 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:47.484453 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c2cdb0b40b7e34a78e15c10ce1b8c3ae7f086411e9b61f46fb9068ffb2b7c6d" Jul 22 09:22:49.601164 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:49.601130 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:22:50.221232 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:50.221201 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcsgw_dc8d8789-74a0-44ec-8902-44a9650c41ac/ovn-acl-logging/0.log" Jul 22 09:22:50.223123 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:50.223097 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcsgw_dc8d8789-74a0-44ec-8902-44a9650c41ac/ovn-acl-logging/0.log" Jul 22 09:22:50.224269 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:50.224253 2549 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 09:22:51.064550 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:51.064513 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:22:51.568776 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:51.568737 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8qfss"] Jul 22 09:22:51.569001 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:51.568987 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a" containerName="util" Jul 22 09:22:51.569042 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:51.569003 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a" containerName="util" Jul 22 09:22:51.569042 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:51.569021 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a" containerName="pull" Jul 22 09:22:51.569042 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:51.569026 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a" containerName="pull" Jul 22 09:22:51.569042 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:51.569032 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a" containerName="extract" Jul 22 09:22:51.569042 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:51.569038 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a" containerName="extract" Jul 22 09:22:51.569191 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:51.569087 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f2e53b2-62bc-4bf0-8cda-ca4f0d2c300a" containerName="extract" Jul 22 09:22:51.571769 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:51.571753 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8qfss" Jul 22 09:22:51.575463 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:51.575446 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-2sbpt\"" Jul 22 09:22:51.575769 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:51.575753 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 22 09:22:51.576507 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:51.576490 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 22 09:22:51.586122 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:51.586096 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8qfss"] Jul 22 09:22:51.659729 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:51.659684 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xdv7\" (UniqueName: \"kubernetes.io/projected/5df337ef-a270-41aa-907c-3a157fd2e5d2-kube-api-access-5xdv7\") pod \"cert-manager-operator-controller-manager-55c444bf86-8qfss\" (UID: \"5df337ef-a270-41aa-907c-3a157fd2e5d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8qfss" Jul 22 09:22:51.659903 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:51.659795 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5df337ef-a270-41aa-907c-3a157fd2e5d2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8qfss\" (UID: \"5df337ef-a270-41aa-907c-3a157fd2e5d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8qfss" Jul 22 09:22:51.760221 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:51.760182 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xdv7\" (UniqueName: \"kubernetes.io/projected/5df337ef-a270-41aa-907c-3a157fd2e5d2-kube-api-access-5xdv7\") pod \"cert-manager-operator-controller-manager-55c444bf86-8qfss\" (UID: \"5df337ef-a270-41aa-907c-3a157fd2e5d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8qfss" Jul 22 09:22:51.760394 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:51.760251 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5df337ef-a270-41aa-907c-3a157fd2e5d2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8qfss\" (UID: \"5df337ef-a270-41aa-907c-3a157fd2e5d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8qfss" Jul 22 09:22:51.760617 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:51.760600 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5df337ef-a270-41aa-907c-3a157fd2e5d2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8qfss\" (UID: \"5df337ef-a270-41aa-907c-3a157fd2e5d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8qfss" Jul 22 09:22:51.769477 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:51.769451 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xdv7\" (UniqueName: \"kubernetes.io/projected/5df337ef-a270-41aa-907c-3a157fd2e5d2-kube-api-access-5xdv7\") pod \"cert-manager-operator-controller-manager-55c444bf86-8qfss\" (UID: \"5df337ef-a270-41aa-907c-3a157fd2e5d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8qfss" Jul 22 09:22:51.880491 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:51.880404 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8qfss" Jul 22 09:22:52.019855 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:52.019768 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8qfss"] Jul 22 09:22:52.023155 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:22:52.023131 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df337ef_a270_41aa_907c_3a157fd2e5d2.slice/crio-083035e293154456a227c08e7d353ba1154021c5b429f3c191ef5636d2e1e866 WatchSource:0}: Error finding container 083035e293154456a227c08e7d353ba1154021c5b429f3c191ef5636d2e1e866: Status 404 returned error can't find the container with id 083035e293154456a227c08e7d353ba1154021c5b429f3c191ef5636d2e1e866 Jul 22 09:22:52.503819 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:52.503778 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8qfss" event={"ID":"5df337ef-a270-41aa-907c-3a157fd2e5d2","Type":"ContainerStarted","Data":"083035e293154456a227c08e7d353ba1154021c5b429f3c191ef5636d2e1e866"} Jul 22 09:22:53.213751 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:53.213701 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:22:53.309467 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:53.309433 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:22:53.509634 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:53.509538 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8qfss" event={"ID":"5df337ef-a270-41aa-907c-3a157fd2e5d2","Type":"ContainerStarted","Data":"77058e525fa4fd8d52142cd2f80791a3cbbc45f60eea5a8c9399865306c295c5"} Jul 22 09:22:53.533810 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:53.533759 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8qfss" podStartSLOduration=1.258974093 podStartE2EDuration="2.533737849s" totalImagesPullingTime="1.274763756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:22:53.530967137 +0000 UTC m=+303.994761575" watchObservedRunningTime="2026-07-22 09:22:53.533737849 +0000 UTC m=+303.997532286" Jul 22 09:22:53.908028 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:53.907987 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:22:54.007955 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:54.007921 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:22:54.901758 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:54.901720 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v"] Jul 22 09:22:54.905338 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:54.905315 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v" Jul 22 09:22:54.908146 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:54.908109 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 09:22:54.908256 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:54.908228 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gjwg9\"" Jul 22 09:22:54.908343 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:54.908276 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 09:22:54.914166 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:54.914145 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v"] Jul 22 09:22:54.985283 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:54.985248 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v\" (UID: \"c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v" Jul 22 09:22:54.985283 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:54.985284 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88t7c\" (UniqueName: \"kubernetes.io/projected/c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2-kube-api-access-88t7c\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v\" (UID: \"c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v" Jul 22 09:22:54.985481 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:54.985330 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v\" (UID: \"c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v" Jul 22 09:22:55.011294 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.011255 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:22:55.086632 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.086599 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v\" (UID: \"c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v" Jul 22 09:22:55.086784 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.086656 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v\" (UID: \"c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v" Jul 22 09:22:55.086784 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.086691 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88t7c\" (UniqueName: \"kubernetes.io/projected/c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2-kube-api-access-88t7c\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v\" (UID: \"c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v" Jul 22 09:22:55.087070 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.087054 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v\" (UID: \"c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v" Jul 22 09:22:55.087109 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.087071 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v\" (UID: \"c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v" Jul 22 09:22:55.105217 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.105184 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88t7c\" (UniqueName: \"kubernetes.io/projected/c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2-kube-api-access-88t7c\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v\" (UID: \"c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v" Jul 22 09:22:55.140908 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.140876 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:22:55.215060 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.214948 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v" Jul 22 09:22:55.353055 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.353030 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v"] Jul 22 09:22:55.355398 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:22:55.355369 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24e2659_cdd4_4b9a_a4e1_bdf47075cfa2.slice/crio-986a1a5fa72109396afbefa5c76e3dcac0256ee8389810ccbfeaedf84a00cd14 WatchSource:0}: Error finding container 986a1a5fa72109396afbefa5c76e3dcac0256ee8389810ccbfeaedf84a00cd14: Status 404 returned error can't find the container with id 986a1a5fa72109396afbefa5c76e3dcac0256ee8389810ccbfeaedf84a00cd14 Jul 22 09:22:55.359702 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.359678 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-v67f8"] Jul 22 09:22:55.362895 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.362878 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-v67f8" Jul 22 09:22:55.365406 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.365379 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6j8hx\"" Jul 22 09:22:55.365632 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.365613 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 22 09:22:55.365709 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.365690 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 22 09:22:55.373991 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.373965 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-v67f8"] Jul 22 09:22:55.490367 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.490281 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc022df6-3ba3-4596-b2e2-cecbf006169c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-v67f8\" (UID: \"bc022df6-3ba3-4596-b2e2-cecbf006169c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v67f8" Jul 22 09:22:55.490367 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.490329 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqwkb\" (UniqueName: \"kubernetes.io/projected/bc022df6-3ba3-4596-b2e2-cecbf006169c-kube-api-access-qqwkb\") pod \"cert-manager-webhook-6486ffd997-v67f8\" (UID: \"bc022df6-3ba3-4596-b2e2-cecbf006169c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v67f8" Jul 22 09:22:55.518167 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.518132 2549 generic.go:356] "Generic (PLEG): container finished" podID="c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2" containerID="7ce75bd9b19304298e1094b222099cc92ad51256cd5ddd8b448ba79983219911" exitCode=0 Jul 22 09:22:55.518287 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.518215 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v" event={"ID":"c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2","Type":"ContainerDied","Data":"7ce75bd9b19304298e1094b222099cc92ad51256cd5ddd8b448ba79983219911"} Jul 22 09:22:55.518287 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.518268 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v" event={"ID":"c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2","Type":"ContainerStarted","Data":"986a1a5fa72109396afbefa5c76e3dcac0256ee8389810ccbfeaedf84a00cd14"} Jul 22 09:22:55.519147 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.519133 2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 09:22:55.591070 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.591037 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc022df6-3ba3-4596-b2e2-cecbf006169c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-v67f8\" (UID: \"bc022df6-3ba3-4596-b2e2-cecbf006169c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v67f8" Jul 22 09:22:55.591070 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.591071 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqwkb\" (UniqueName: \"kubernetes.io/projected/bc022df6-3ba3-4596-b2e2-cecbf006169c-kube-api-access-qqwkb\") pod \"cert-manager-webhook-6486ffd997-v67f8\" (UID: \"bc022df6-3ba3-4596-b2e2-cecbf006169c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v67f8" Jul 22 09:22:55.599927 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.599890 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc022df6-3ba3-4596-b2e2-cecbf006169c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-v67f8\" (UID: \"bc022df6-3ba3-4596-b2e2-cecbf006169c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v67f8" Jul 22 09:22:55.600044 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.600011 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqwkb\" (UniqueName: \"kubernetes.io/projected/bc022df6-3ba3-4596-b2e2-cecbf006169c-kube-api-access-qqwkb\") pod \"cert-manager-webhook-6486ffd997-v67f8\" (UID: \"bc022df6-3ba3-4596-b2e2-cecbf006169c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v67f8" Jul 22 09:22:55.699935 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.699902 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-v67f8" Jul 22 09:22:55.823075 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:55.823035 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-v67f8"] Jul 22 09:22:55.826028 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:22:55.826004 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc022df6_3ba3_4596_b2e2_cecbf006169c.slice/crio-e55c4e3f0224b74af9b09796ea3a1a8eba3040fa6f4074aba0502d589bac4317 WatchSource:0}: Error finding container e55c4e3f0224b74af9b09796ea3a1a8eba3040fa6f4074aba0502d589bac4317: Status 404 returned error can't find the container with id e55c4e3f0224b74af9b09796ea3a1a8eba3040fa6f4074aba0502d589bac4317 Jul 22 09:22:56.527942 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:56.527909 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-v67f8" event={"ID":"bc022df6-3ba3-4596-b2e2-cecbf006169c","Type":"ContainerStarted","Data":"e55c4e3f0224b74af9b09796ea3a1a8eba3040fa6f4074aba0502d589bac4317"} Jul 22 09:22:57.219148 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:57.219065 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:22:57.408237 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:57.408194 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:22:59.542295 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:59.542258 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-v67f8" event={"ID":"bc022df6-3ba3-4596-b2e2-cecbf006169c","Type":"ContainerStarted","Data":"48c8a2f1bdfd895f3c20c8a7e3d92792eb1ff50f5bee3c410bfc3876b0546a8a"} Jul 22 09:22:59.542779 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:59.542493 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-v67f8" Jul 22 09:22:59.543996 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:59.543972 2549 generic.go:356] "Generic (PLEG): container finished" podID="c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2" containerID="74f7ffe4988f601d0c4fe55f4b442c276723424067c5edc338e607474ce61eae" exitCode=0 Jul 22 09:22:59.544095 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:59.544004 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v" event={"ID":"c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2","Type":"ContainerDied","Data":"74f7ffe4988f601d0c4fe55f4b442c276723424067c5edc338e607474ce61eae"} Jul 22 09:22:59.561439 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:22:59.561396 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-v67f8" podStartSLOduration=1.766078647 podStartE2EDuration="4.561377871s" totalImagesPullingTime="2.795299224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:22:59.559943195 +0000 UTC m=+310.023737633" watchObservedRunningTime="2026-07-22 09:22:59.561377871 +0000 UTC m=+310.025172310" Jul 22 09:23:00.416370 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:00.416334 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:00.529024 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:00.528988 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:00.550074 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:00.550046 2549 generic.go:356] "Generic (PLEG): container finished" podID="c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2" containerID="b467c3636ac21383013dfab56d48e856cb20ed35a97bc7d530f557a2a5057d05" exitCode=0 Jul 22 09:23:00.550533 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:00.550136 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v" event={"ID":"c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2","Type":"ContainerDied","Data":"b467c3636ac21383013dfab56d48e856cb20ed35a97bc7d530f557a2a5057d05"} Jul 22 09:23:01.672326 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:01.672288 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v" Jul 22 09:23:01.739969 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:01.739939 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88t7c\" (UniqueName: \"kubernetes.io/projected/c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2-kube-api-access-88t7c\") pod \"c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2\" (UID: \"c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2\") " Jul 22 09:23:01.740111 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:01.739982 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2-util\") pod \"c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2\" (UID: \"c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2\") " Jul 22 09:23:01.740111 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:01.740026 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2-bundle\") pod \"c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2\" (UID: \"c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2\") " Jul 22 09:23:01.740431 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:01.740402 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2-bundle" (OuterVolumeSpecName: "bundle") pod "c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2" (UID: "c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:23:01.741971 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:01.741949 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2-kube-api-access-88t7c" (OuterVolumeSpecName: "kube-api-access-88t7c") pod "c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2" (UID: "c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2"). InnerVolumeSpecName "kube-api-access-88t7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:23:01.743931 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:01.743909 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2-util" (OuterVolumeSpecName: "util") pod "c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2" (UID: "c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:23:01.840813 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:01.840720 2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2-bundle\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:23:01.840813 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:01.840760 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88t7c\" (UniqueName: \"kubernetes.io/projected/c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2-kube-api-access-88t7c\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:23:01.840813 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:01.840771 2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2-util\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:23:02.163317 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:02.163224 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xqtl5"] Jul 22 09:23:02.163535 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:02.163522 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2" containerName="util" Jul 22 09:23:02.163585 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:02.163536 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2" containerName="util" Jul 22 09:23:02.163585 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:02.163548 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2" containerName="pull" Jul 22 09:23:02.163585 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:02.163553 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2" containerName="pull" Jul 22 09:23:02.163585 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:02.163566 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2" containerName="extract" Jul 22 09:23:02.163585 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:02.163571 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2" containerName="extract" Jul 22 09:23:02.163732 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:02.163610 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2" containerName="extract" Jul 22 09:23:02.167644 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:02.167627 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xqtl5" Jul 22 09:23:02.170127 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:02.170109 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-9swsv\"" Jul 22 09:23:02.176804 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:02.176781 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xqtl5"] Jul 22 09:23:02.244132 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:02.244099 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8bc69630-f736-472b-9929-aa4040fa78e3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xqtl5\" (UID: \"8bc69630-f736-472b-9929-aa4040fa78e3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xqtl5" Jul 22 09:23:02.244132 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:02.244143 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcl4p\" (UniqueName: \"kubernetes.io/projected/8bc69630-f736-472b-9929-aa4040fa78e3-kube-api-access-rcl4p\") pod \"cert-manager-cainjector-559cdd8c87-xqtl5\" (UID: \"8bc69630-f736-472b-9929-aa4040fa78e3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xqtl5" Jul 22 09:23:02.344612 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:02.344582 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcl4p\" (UniqueName: \"kubernetes.io/projected/8bc69630-f736-472b-9929-aa4040fa78e3-kube-api-access-rcl4p\") pod \"cert-manager-cainjector-559cdd8c87-xqtl5\" (UID: \"8bc69630-f736-472b-9929-aa4040fa78e3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xqtl5" Jul 22 09:23:02.344788 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:02.344645 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8bc69630-f736-472b-9929-aa4040fa78e3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xqtl5\" (UID: \"8bc69630-f736-472b-9929-aa4040fa78e3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xqtl5" Jul 22 09:23:02.353212 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:02.353183 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8bc69630-f736-472b-9929-aa4040fa78e3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xqtl5\" (UID: \"8bc69630-f736-472b-9929-aa4040fa78e3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xqtl5" Jul 22 09:23:02.353600 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:02.353580 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcl4p\" (UniqueName: \"kubernetes.io/projected/8bc69630-f736-472b-9929-aa4040fa78e3-kube-api-access-rcl4p\") pod \"cert-manager-cainjector-559cdd8c87-xqtl5\" (UID: \"8bc69630-f736-472b-9929-aa4040fa78e3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xqtl5" Jul 22 09:23:02.475912 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:02.475825 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xqtl5" Jul 22 09:23:02.562013 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:02.561972 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v" Jul 22 09:23:02.562275 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:02.561969 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftss8v" event={"ID":"c24e2659-cdd4-4b9a-a4e1-bdf47075cfa2","Type":"ContainerDied","Data":"986a1a5fa72109396afbefa5c76e3dcac0256ee8389810ccbfeaedf84a00cd14"} Jul 22 09:23:02.562275 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:02.562247 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="986a1a5fa72109396afbefa5c76e3dcac0256ee8389810ccbfeaedf84a00cd14" Jul 22 09:23:02.629850 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:02.629812 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xqtl5"] Jul 22 09:23:02.633713 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:23:02.633683 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bc69630_f736_472b_9929_aa4040fa78e3.slice/crio-15f96ed21e0ed561a2c55a52afa9264fb2f35aab3172bd2cfc620252782efc34 WatchSource:0}: Error finding container 15f96ed21e0ed561a2c55a52afa9264fb2f35aab3172bd2cfc620252782efc34: Status 404 returned error can't find the container with id 15f96ed21e0ed561a2c55a52afa9264fb2f35aab3172bd2cfc620252782efc34 Jul 22 09:23:03.420182 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:03.420148 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:03.482165 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:03.482126 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:03.567590 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:03.567551 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xqtl5" event={"ID":"8bc69630-f736-472b-9929-aa4040fa78e3","Type":"ContainerStarted","Data":"2658d15f58cd5572dbcaa4529ea0a6a2d992230b8fb2e4859265399489ce605a"} Jul 22 09:23:03.567778 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:03.567600 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xqtl5" event={"ID":"8bc69630-f736-472b-9929-aa4040fa78e3","Type":"ContainerStarted","Data":"15f96ed21e0ed561a2c55a52afa9264fb2f35aab3172bd2cfc620252782efc34"} Jul 22 09:23:03.586981 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:03.586928 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xqtl5" podStartSLOduration=1.586911727 podStartE2EDuration="1.586911727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:23:03.585718785 +0000 UTC m=+314.049513222" watchObservedRunningTime="2026-07-22 09:23:03.586911727 +0000 UTC m=+314.050706166" Jul 22 09:23:05.450771 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:05.450740 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:05.552572 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:05.552546 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-v67f8" Jul 22 09:23:08.608446 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:08.608417 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:08.712468 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:08.712430 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:14.183069 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:14.183025 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:14.261467 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:14.261436 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:16.110846 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:16.110805 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t"] Jul 22 09:23:16.118177 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:16.118149 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t" Jul 22 09:23:16.123797 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:16.123770 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 09:23:16.124413 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:16.124390 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 09:23:16.124558 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:16.124465 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gjwg9\"" Jul 22 09:23:16.126839 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:16.126815 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t"] Jul 22 09:23:16.252156 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:16.252117 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phx2k\" (UniqueName: \"kubernetes.io/projected/fd790e66-faf6-499c-ace4-ff8c73f741ec-kube-api-access-phx2k\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t\" (UID: \"fd790e66-faf6-499c-ace4-ff8c73f741ec\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t" Jul 22 09:23:16.252366 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:16.252186 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd790e66-faf6-499c-ace4-ff8c73f741ec-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t\" (UID: \"fd790e66-faf6-499c-ace4-ff8c73f741ec\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t" Jul 22 09:23:16.252366 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:16.252240 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd790e66-faf6-499c-ace4-ff8c73f741ec-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t\" (UID: \"fd790e66-faf6-499c-ace4-ff8c73f741ec\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t" Jul 22 09:23:16.352620 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:16.352589 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phx2k\" (UniqueName: \"kubernetes.io/projected/fd790e66-faf6-499c-ace4-ff8c73f741ec-kube-api-access-phx2k\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t\" (UID: \"fd790e66-faf6-499c-ace4-ff8c73f741ec\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t" Jul 22 09:23:16.352806 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:16.352639 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd790e66-faf6-499c-ace4-ff8c73f741ec-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t\" (UID: \"fd790e66-faf6-499c-ace4-ff8c73f741ec\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t" Jul 22 09:23:16.352806 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:16.352678 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd790e66-faf6-499c-ace4-ff8c73f741ec-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t\" (UID: \"fd790e66-faf6-499c-ace4-ff8c73f741ec\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t" Jul 22 09:23:16.353068 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:16.353041 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd790e66-faf6-499c-ace4-ff8c73f741ec-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t\" (UID: \"fd790e66-faf6-499c-ace4-ff8c73f741ec\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t" Jul 22 09:23:16.353135 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:16.353075 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd790e66-faf6-499c-ace4-ff8c73f741ec-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t\" (UID: \"fd790e66-faf6-499c-ace4-ff8c73f741ec\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t" Jul 22 09:23:16.361979 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:16.361916 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phx2k\" (UniqueName: \"kubernetes.io/projected/fd790e66-faf6-499c-ace4-ff8c73f741ec-kube-api-access-phx2k\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t\" (UID: \"fd790e66-faf6-499c-ace4-ff8c73f741ec\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t" Jul 22 09:23:16.396188 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:16.396154 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:16.432004 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:16.431976 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t" Jul 22 09:23:16.558600 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:16.558568 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t"] Jul 22 09:23:16.561586 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:23:16.561560 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd790e66_faf6_499c_ace4_ff8c73f741ec.slice/crio-0f766e25aa61aeae1143df72bbf5a2e20e7f1cb20d7a7e613203d5cc31e2eb74 WatchSource:0}: Error finding container 0f766e25aa61aeae1143df72bbf5a2e20e7f1cb20d7a7e613203d5cc31e2eb74: Status 404 returned error can't find the container with id 0f766e25aa61aeae1143df72bbf5a2e20e7f1cb20d7a7e613203d5cc31e2eb74 Jul 22 09:23:16.618430 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:16.618405 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t" event={"ID":"fd790e66-faf6-499c-ace4-ff8c73f741ec","Type":"ContainerStarted","Data":"0f766e25aa61aeae1143df72bbf5a2e20e7f1cb20d7a7e613203d5cc31e2eb74"} Jul 22 09:23:17.624458 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:17.624375 2549 generic.go:356] "Generic (PLEG): container finished" podID="fd790e66-faf6-499c-ace4-ff8c73f741ec" containerID="5b033483a3f3180265f77677b07df7a80cbd771006819cfb6ecb78421e5db79a" exitCode=0 Jul 22 09:23:17.624458 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:17.624435 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t" event={"ID":"fd790e66-faf6-499c-ace4-ff8c73f741ec","Type":"ContainerDied","Data":"5b033483a3f3180265f77677b07df7a80cbd771006819cfb6ecb78421e5db79a"} Jul 22 09:23:19.634555 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:19.634511 2549 generic.go:356] "Generic (PLEG): container finished" podID="fd790e66-faf6-499c-ace4-ff8c73f741ec" containerID="65c390138b878fecd59b8081b85fc77f3a1d0ff6e2767b7f98fc5e07c63bf03d" exitCode=0 Jul 22 09:23:19.634555 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:19.634542 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t" event={"ID":"fd790e66-faf6-499c-ace4-ff8c73f741ec","Type":"ContainerDied","Data":"65c390138b878fecd59b8081b85fc77f3a1d0ff6e2767b7f98fc5e07c63bf03d"} Jul 22 09:23:20.640660 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:20.640624 2549 generic.go:356] "Generic (PLEG): container finished" podID="fd790e66-faf6-499c-ace4-ff8c73f741ec" containerID="27a1c94c3c909237974acd95b11477462e966e934df93cef08071c7375c2c928" exitCode=0 Jul 22 09:23:20.640660 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:20.640666 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t" event={"ID":"fd790e66-faf6-499c-ace4-ff8c73f741ec","Type":"ContainerDied","Data":"27a1c94c3c909237974acd95b11477462e966e934df93cef08071c7375c2c928"} Jul 22 09:23:21.766916 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:21.766892 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t" Jul 22 09:23:21.893690 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:21.893658 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd790e66-faf6-499c-ace4-ff8c73f741ec-bundle\") pod \"fd790e66-faf6-499c-ace4-ff8c73f741ec\" (UID: \"fd790e66-faf6-499c-ace4-ff8c73f741ec\") " Jul 22 09:23:21.893847 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:21.893745 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd790e66-faf6-499c-ace4-ff8c73f741ec-util\") pod \"fd790e66-faf6-499c-ace4-ff8c73f741ec\" (UID: \"fd790e66-faf6-499c-ace4-ff8c73f741ec\") " Jul 22 09:23:21.893847 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:21.893765 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phx2k\" (UniqueName: \"kubernetes.io/projected/fd790e66-faf6-499c-ace4-ff8c73f741ec-kube-api-access-phx2k\") pod \"fd790e66-faf6-499c-ace4-ff8c73f741ec\" (UID: \"fd790e66-faf6-499c-ace4-ff8c73f741ec\") " Jul 22 09:23:21.894560 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:21.894533 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd790e66-faf6-499c-ace4-ff8c73f741ec-bundle" (OuterVolumeSpecName: "bundle") pod "fd790e66-faf6-499c-ace4-ff8c73f741ec" (UID: "fd790e66-faf6-499c-ace4-ff8c73f741ec"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:23:21.895713 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:21.895662 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd790e66-faf6-499c-ace4-ff8c73f741ec-kube-api-access-phx2k" (OuterVolumeSpecName: "kube-api-access-phx2k") pod "fd790e66-faf6-499c-ace4-ff8c73f741ec" (UID: "fd790e66-faf6-499c-ace4-ff8c73f741ec"). InnerVolumeSpecName "kube-api-access-phx2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:23:21.898648 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:21.898626 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd790e66-faf6-499c-ace4-ff8c73f741ec-util" (OuterVolumeSpecName: "util") pod "fd790e66-faf6-499c-ace4-ff8c73f741ec" (UID: "fd790e66-faf6-499c-ace4-ff8c73f741ec"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:23:21.995186 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:21.995150 2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd790e66-faf6-499c-ace4-ff8c73f741ec-util\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:23:21.995186 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:21.995182 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phx2k\" (UniqueName: \"kubernetes.io/projected/fd790e66-faf6-499c-ace4-ff8c73f741ec-kube-api-access-phx2k\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:23:21.995186 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:21.995194 2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd790e66-faf6-499c-ace4-ff8c73f741ec-bundle\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:23:22.612707 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.612620 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:22.651720 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.651695 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t" Jul 22 09:23:22.651720 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.651708 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftnf2t" event={"ID":"fd790e66-faf6-499c-ace4-ff8c73f741ec","Type":"ContainerDied","Data":"0f766e25aa61aeae1143df72bbf5a2e20e7f1cb20d7a7e613203d5cc31e2eb74"} Jul 22 09:23:22.651894 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.651757 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f766e25aa61aeae1143df72bbf5a2e20e7f1cb20d7a7e613203d5cc31e2eb74" Jul 22 09:23:22.682280 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.682255 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d9cd798d9-f4jnw"] Jul 22 09:23:22.682530 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.682518 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd790e66-faf6-499c-ace4-ff8c73f741ec" containerName="pull" Jul 22 09:23:22.682576 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.682532 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd790e66-faf6-499c-ace4-ff8c73f741ec" containerName="pull" Jul 22 09:23:22.682576 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.682552 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd790e66-faf6-499c-ace4-ff8c73f741ec" containerName="util" Jul 22 09:23:22.682576 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.682557 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd790e66-faf6-499c-ace4-ff8c73f741ec" containerName="util" Jul 22 09:23:22.682576 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.682562 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd790e66-faf6-499c-ace4-ff8c73f741ec" containerName="extract" Jul 22 09:23:22.682576 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.682568 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd790e66-faf6-499c-ace4-ff8c73f741ec" containerName="extract" Jul 22 09:23:22.682721 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.682614 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd790e66-faf6-499c-ace4-ff8c73f741ec" containerName="extract" Jul 22 09:23:22.687073 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.687057 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-f4jnw" Jul 22 09:23:22.690006 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.689986 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-nnm2t\"" Jul 22 09:23:22.690108 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.689993 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 22 09:23:22.690165 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.690110 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 22 09:23:22.690782 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.690763 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 22 09:23:22.690880 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.690772 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 22 09:23:22.690880 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.690821 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 22 09:23:22.695060 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.695042 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d9cd798d9-f4jnw"] Jul 22 09:23:22.704041 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.704011 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:22.801355 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.801326 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1467962b-93b6-45dc-bf90-3715e9c078c4-cert\") pod \"lws-controller-manager-7d9cd798d9-f4jnw\" (UID: \"1467962b-93b6-45dc-bf90-3715e9c078c4\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-f4jnw" Jul 22 09:23:22.801669 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.801361 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1467962b-93b6-45dc-bf90-3715e9c078c4-metrics-cert\") pod \"lws-controller-manager-7d9cd798d9-f4jnw\" (UID: \"1467962b-93b6-45dc-bf90-3715e9c078c4\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-f4jnw" Jul 22 09:23:22.801669 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.801428 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7449z\" (UniqueName: \"kubernetes.io/projected/1467962b-93b6-45dc-bf90-3715e9c078c4-kube-api-access-7449z\") pod \"lws-controller-manager-7d9cd798d9-f4jnw\" (UID: \"1467962b-93b6-45dc-bf90-3715e9c078c4\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-f4jnw" Jul 22 09:23:22.801669 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.801473 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1467962b-93b6-45dc-bf90-3715e9c078c4-manager-config\") pod \"lws-controller-manager-7d9cd798d9-f4jnw\" (UID: \"1467962b-93b6-45dc-bf90-3715e9c078c4\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-f4jnw" Jul 22 09:23:22.902511 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.902435 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7449z\" (UniqueName: \"kubernetes.io/projected/1467962b-93b6-45dc-bf90-3715e9c078c4-kube-api-access-7449z\") pod \"lws-controller-manager-7d9cd798d9-f4jnw\" (UID: \"1467962b-93b6-45dc-bf90-3715e9c078c4\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-f4jnw" Jul 22 09:23:22.902511 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.902472 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1467962b-93b6-45dc-bf90-3715e9c078c4-manager-config\") pod \"lws-controller-manager-7d9cd798d9-f4jnw\" (UID: \"1467962b-93b6-45dc-bf90-3715e9c078c4\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-f4jnw" Jul 22 09:23:22.902665 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.902581 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1467962b-93b6-45dc-bf90-3715e9c078c4-cert\") pod \"lws-controller-manager-7d9cd798d9-f4jnw\" (UID: \"1467962b-93b6-45dc-bf90-3715e9c078c4\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-f4jnw" Jul 22 09:23:22.902665 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.902603 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1467962b-93b6-45dc-bf90-3715e9c078c4-metrics-cert\") pod \"lws-controller-manager-7d9cd798d9-f4jnw\" (UID: \"1467962b-93b6-45dc-bf90-3715e9c078c4\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-f4jnw" Jul 22 09:23:22.903244 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.903218 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1467962b-93b6-45dc-bf90-3715e9c078c4-manager-config\") pod \"lws-controller-manager-7d9cd798d9-f4jnw\" (UID: \"1467962b-93b6-45dc-bf90-3715e9c078c4\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-f4jnw" Jul 22 09:23:22.904971 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.904952 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1467962b-93b6-45dc-bf90-3715e9c078c4-metrics-cert\") pod \"lws-controller-manager-7d9cd798d9-f4jnw\" (UID: \"1467962b-93b6-45dc-bf90-3715e9c078c4\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-f4jnw" Jul 22 09:23:22.905104 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.905087 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1467962b-93b6-45dc-bf90-3715e9c078c4-cert\") pod \"lws-controller-manager-7d9cd798d9-f4jnw\" (UID: \"1467962b-93b6-45dc-bf90-3715e9c078c4\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-f4jnw" Jul 22 09:23:22.912569 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.912549 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7449z\" (UniqueName: \"kubernetes.io/projected/1467962b-93b6-45dc-bf90-3715e9c078c4-kube-api-access-7449z\") pod \"lws-controller-manager-7d9cd798d9-f4jnw\" (UID: \"1467962b-93b6-45dc-bf90-3715e9c078c4\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-f4jnw" Jul 22 09:23:22.996163 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:22.996130 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-f4jnw" Jul 22 09:23:23.131156 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:23.131122 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d9cd798d9-f4jnw"] Jul 22 09:23:23.134148 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:23:23.134124 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1467962b_93b6_45dc_bf90_3715e9c078c4.slice/crio-05d4bca6534cc2c7c1f4cc4f1cafc062b260ff38a8a676b3cdcc0a979c0fa4c0 WatchSource:0}: Error finding container 05d4bca6534cc2c7c1f4cc4f1cafc062b260ff38a8a676b3cdcc0a979c0fa4c0: Status 404 returned error can't find the container with id 05d4bca6534cc2c7c1f4cc4f1cafc062b260ff38a8a676b3cdcc0a979c0fa4c0 Jul 22 09:23:23.658570 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:23.658534 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-f4jnw" event={"ID":"1467962b-93b6-45dc-bf90-3715e9c078c4","Type":"ContainerStarted","Data":"05d4bca6534cc2c7c1f4cc4f1cafc062b260ff38a8a676b3cdcc0a979c0fa4c0"} Jul 22 09:23:25.672044 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:25.672010 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-f4jnw" event={"ID":"1467962b-93b6-45dc-bf90-3715e9c078c4","Type":"ContainerStarted","Data":"f60fd1f60b91b5e1df3a77545601fd6c38d1c7f048c39da1add1cd1694857438"} Jul 22 09:23:25.672442 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:25.672081 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-f4jnw" Jul 22 09:23:25.691682 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:25.691635 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-f4jnw" podStartSLOduration=1.912390728 podStartE2EDuration="3.691621653s" totalImagesPullingTime="1.779230925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:23:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:23:25.691229488 +0000 UTC m=+336.155023926" watchObservedRunningTime="2026-07-22 09:23:25.691621653 +0000 UTC m=+336.155416087" Jul 22 09:23:31.801083 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:31.801048 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:31.900389 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:31.900357 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:32.414768 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:32.414731 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:32.648237 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:32.648209 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:32.755608 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:32.755524 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:32.780215 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:32.780185 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9"] Jul 22 09:23:32.824044 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:32.824009 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9"] Jul 22 09:23:32.824446 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:32.824130 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" Jul 22 09:23:32.827155 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:32.827068 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 09:23:32.827155 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:32.827080 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fx9bz\"" Jul 22 09:23:32.827155 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:32.827120 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 09:23:32.827384 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:32.827123 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 09:23:32.827384 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:32.827203 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 09:23:32.909276 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:32.909222 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:32.985323 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:32.985277 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/29755782-61fa-4b59-9063-3b06f072620b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67797c6b-l7dw9\" (UID: \"29755782-61fa-4b59-9063-3b06f072620b\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" Jul 22 09:23:32.985513 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:32.985346 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/29755782-61fa-4b59-9063-3b06f072620b-manifests\") pod \"opendatahub-operator-controller-manager-67797c6b-l7dw9\" (UID: \"29755782-61fa-4b59-9063-3b06f072620b\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" Jul 22 09:23:32.985513 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:32.985432 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29755782-61fa-4b59-9063-3b06f072620b-tmp\") pod \"opendatahub-operator-controller-manager-67797c6b-l7dw9\" (UID: \"29755782-61fa-4b59-9063-3b06f072620b\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" Jul 22 09:23:32.985513 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:32.985487 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x75rw\" (UniqueName: \"kubernetes.io/projected/29755782-61fa-4b59-9063-3b06f072620b-kube-api-access-x75rw\") pod \"opendatahub-operator-controller-manager-67797c6b-l7dw9\" (UID: \"29755782-61fa-4b59-9063-3b06f072620b\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" Jul 22 09:23:32.985655 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:32.985527 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/29755782-61fa-4b59-9063-3b06f072620b-webhook-cert\") pod \"opendatahub-operator-controller-manager-67797c6b-l7dw9\" (UID: \"29755782-61fa-4b59-9063-3b06f072620b\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" Jul 22 09:23:33.012091 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:33.012015 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:33.086170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:33.086137 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29755782-61fa-4b59-9063-3b06f072620b-tmp\") pod \"opendatahub-operator-controller-manager-67797c6b-l7dw9\" (UID: \"29755782-61fa-4b59-9063-3b06f072620b\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" Jul 22 09:23:33.086347 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:33.086180 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x75rw\" (UniqueName: \"kubernetes.io/projected/29755782-61fa-4b59-9063-3b06f072620b-kube-api-access-x75rw\") pod \"opendatahub-operator-controller-manager-67797c6b-l7dw9\" (UID: \"29755782-61fa-4b59-9063-3b06f072620b\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" Jul 22 09:23:33.086347 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:33.086205 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/29755782-61fa-4b59-9063-3b06f072620b-webhook-cert\") pod \"opendatahub-operator-controller-manager-67797c6b-l7dw9\" (UID: \"29755782-61fa-4b59-9063-3b06f072620b\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" Jul 22 09:23:33.086347 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:33.086242 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/29755782-61fa-4b59-9063-3b06f072620b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67797c6b-l7dw9\" (UID: \"29755782-61fa-4b59-9063-3b06f072620b\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" Jul 22 09:23:33.086488 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:33.086348 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/29755782-61fa-4b59-9063-3b06f072620b-manifests\") pod \"opendatahub-operator-controller-manager-67797c6b-l7dw9\" (UID: \"29755782-61fa-4b59-9063-3b06f072620b\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" Jul 22 09:23:33.086685 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:33.086660 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/29755782-61fa-4b59-9063-3b06f072620b-manifests\") pod \"opendatahub-operator-controller-manager-67797c6b-l7dw9\" (UID: \"29755782-61fa-4b59-9063-3b06f072620b\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" Jul 22 09:23:33.088606 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:33.088587 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29755782-61fa-4b59-9063-3b06f072620b-tmp\") pod \"opendatahub-operator-controller-manager-67797c6b-l7dw9\" (UID: \"29755782-61fa-4b59-9063-3b06f072620b\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" Jul 22 09:23:33.088845 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:33.088823 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/29755782-61fa-4b59-9063-3b06f072620b-webhook-cert\") pod \"opendatahub-operator-controller-manager-67797c6b-l7dw9\" (UID: \"29755782-61fa-4b59-9063-3b06f072620b\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" Jul 22 09:23:33.088908 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:33.088854 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/29755782-61fa-4b59-9063-3b06f072620b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67797c6b-l7dw9\" (UID: \"29755782-61fa-4b59-9063-3b06f072620b\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" Jul 22 09:23:33.096004 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:33.095978 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x75rw\" (UniqueName: \"kubernetes.io/projected/29755782-61fa-4b59-9063-3b06f072620b-kube-api-access-x75rw\") pod \"opendatahub-operator-controller-manager-67797c6b-l7dw9\" (UID: \"29755782-61fa-4b59-9063-3b06f072620b\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" Jul 22 09:23:33.112943 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:33.112907 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:33.134724 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:33.134700 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" Jul 22 09:23:33.212337 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:33.212278 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:33.297816 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:33.297788 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9"] Jul 22 09:23:33.301414 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:23:33.301383 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29755782_61fa_4b59_9063_3b06f072620b.slice/crio-88a83cb1cecc8e80a629c5c1eb10f350e409964b1020e62fd3bb83fce275692f WatchSource:0}: Error finding container 88a83cb1cecc8e80a629c5c1eb10f350e409964b1020e62fd3bb83fce275692f: Status 404 returned error can't find the container with id 88a83cb1cecc8e80a629c5c1eb10f350e409964b1020e62fd3bb83fce275692f Jul 22 09:23:33.303062 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:33.303023 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:33.368404 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:33.368368 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:33.469166 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:33.469129 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:33.705040 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:33.704960 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" event={"ID":"29755782-61fa-4b59-9063-3b06f072620b","Type":"ContainerStarted","Data":"88a83cb1cecc8e80a629c5c1eb10f350e409964b1020e62fd3bb83fce275692f"} Jul 22 09:23:36.679288 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:36.679257 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-f4jnw" Jul 22 09:23:36.721881 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:36.721847 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" event={"ID":"29755782-61fa-4b59-9063-3b06f072620b","Type":"ContainerStarted","Data":"8b363e5a650c8426b9013503f639973d91c266e8520b902540acd29f098ab4fd"} Jul 22 09:23:37.727228 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:37.727140 2549 generic.go:356] "Generic (PLEG): container finished" podID="29755782-61fa-4b59-9063-3b06f072620b" containerID="8b363e5a650c8426b9013503f639973d91c266e8520b902540acd29f098ab4fd" exitCode=0 Jul 22 09:23:37.727228 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:37.727198 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" event={"ID":"29755782-61fa-4b59-9063-3b06f072620b","Type":"ContainerDied","Data":"8b363e5a650c8426b9013503f639973d91c266e8520b902540acd29f098ab4fd"} Jul 22 09:23:38.733713 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:38.733625 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" event={"ID":"29755782-61fa-4b59-9063-3b06f072620b","Type":"ContainerStarted","Data":"81f26a06a5fd3271b56040353570a0adf28a108e48cc3fb5d318008afa8027d1"} Jul 22 09:23:38.733713 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:38.733701 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" Jul 22 09:23:38.758579 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:38.758523 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" podStartSLOduration=2.167883331 podStartE2EDuration="6.758508409s" totalImagesPullingTime="3.509072744s" totalInitContainerRuntime="1.081552334s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:23:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:23:38.758038233 +0000 UTC m=+349.221832671" watchObservedRunningTime="2026-07-22 09:23:38.758508409 +0000 UTC m=+349.222302847" Jul 22 09:23:49.742895 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:49.742815 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-l7dw9" Jul 22 09:23:51.307932 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:51.307896 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:51.412616 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:51.412580 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:51.875402 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:51.875294 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:52.324720 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:52.324690 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:52.423013 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:52.422979 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:52.519366 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:52.519335 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:52.627775 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:52.627688 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:52.809000 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:52.808965 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:53.305706 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:53.305676 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:53.446935 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:53.446905 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:53.525602 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:53.525570 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:54.615150 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:54.615118 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:58.503950 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:58.503919 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:23:58.571827 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:23:58.571791 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:24:00.119075 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.119035 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:24:00.216998 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.216967 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:24:00.371404 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.371316 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw"] Jul 22 09:24:00.376982 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.376952 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.379558 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.379536 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-br596\"" Jul 22 09:24:00.379558 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.379553 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 09:24:00.379766 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.379563 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 09:24:00.379766 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.379678 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 09:24:00.388199 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.388171 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw"] Jul 22 09:24:00.399848 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.399810 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:24:00.430852 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.430827 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d"] Jul 22 09:24:00.434736 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.434722 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.438842 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.438816 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 22 09:24:00.438842 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.438829 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-sn5lp\"" Jul 22 09:24:00.438974 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.438820 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 22 09:24:00.438974 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.438875 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 22 09:24:00.451548 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.451524 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d"] Jul 22 09:24:00.478227 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.478198 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.478359 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.478229 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fc9e5acc-1a2a-4dd5-8cc5-800592c516bf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-ggn8d\" (UID: \"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.478359 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.478257 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.478359 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.478285 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc9e5acc-1a2a-4dd5-8cc5-800592c516bf-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-ggn8d\" (UID: \"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.478359 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.478343 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.478507 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.478384 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.478507 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.478400 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc9e5acc-1a2a-4dd5-8cc5-800592c516bf-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-ggn8d\" (UID: \"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.478507 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.478434 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.478507 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.478487 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.478507 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.478504 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.478694 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.478519 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fc9e5acc-1a2a-4dd5-8cc5-800592c516bf-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-ggn8d\" (UID: \"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.478694 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.478548 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40fd4579-0c69-4093-a10d-438cb4f46f09-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.478694 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.478567 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fc9e5acc-1a2a-4dd5-8cc5-800592c516bf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-ggn8d\" (UID: \"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.478694 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.478593 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rngfm\" (UniqueName: \"kubernetes.io/projected/fc9e5acc-1a2a-4dd5-8cc5-800592c516bf-kube-api-access-rngfm\") pod \"istiod-openshift-gateway-94bb8fbfd-ggn8d\" (UID: \"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.478694 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.478619 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl8d5\" (UniqueName: \"kubernetes.io/projected/40fd4579-0c69-4093-a10d-438cb4f46f09-kube-api-access-nl8d5\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.478694 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.478640 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc9e5acc-1a2a-4dd5-8cc5-800592c516bf-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-ggn8d\" (UID: \"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.580011 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.579975 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.580011 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.580016 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc9e5acc-1a2a-4dd5-8cc5-800592c516bf-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-ggn8d\" (UID: \"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.580205 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.580038 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.580205 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.580161 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.580284 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.580208 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc9e5acc-1a2a-4dd5-8cc5-800592c516bf-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-ggn8d\" (UID: \"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.580284 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.580237 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.580420 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.580384 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.580475 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.580438 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.580475 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.580442 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.580475 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.580469 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fc9e5acc-1a2a-4dd5-8cc5-800592c516bf-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-ggn8d\" (UID: \"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.580624 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.580504 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40fd4579-0c69-4093-a10d-438cb4f46f09-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.580624 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.580522 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fc9e5acc-1a2a-4dd5-8cc5-800592c516bf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-ggn8d\" (UID: \"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.580624 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.580556 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rngfm\" (UniqueName: \"kubernetes.io/projected/fc9e5acc-1a2a-4dd5-8cc5-800592c516bf-kube-api-access-rngfm\") pod \"istiod-openshift-gateway-94bb8fbfd-ggn8d\" (UID: \"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.580624 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.580584 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.580624 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.580591 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl8d5\" (UniqueName: \"kubernetes.io/projected/40fd4579-0c69-4093-a10d-438cb4f46f09-kube-api-access-nl8d5\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.580624 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.580615 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc9e5acc-1a2a-4dd5-8cc5-800592c516bf-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-ggn8d\" (UID: \"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.580918 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.580660 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.580918 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.580687 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fc9e5acc-1a2a-4dd5-8cc5-800592c516bf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-ggn8d\" (UID: \"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.580918 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.580767 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.581325 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.581278 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40fd4579-0c69-4093-a10d-438cb4f46f09-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.581477 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.581453 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fc9e5acc-1a2a-4dd5-8cc5-800592c516bf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-ggn8d\" (UID: \"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.581600 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.581579 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.583014 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.582989 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fc9e5acc-1a2a-4dd5-8cc5-800592c516bf-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-ggn8d\" (UID: \"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.583105 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.583023 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc9e5acc-1a2a-4dd5-8cc5-800592c516bf-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-ggn8d\" (UID: \"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.583147 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.583110 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fc9e5acc-1a2a-4dd5-8cc5-800592c516bf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-ggn8d\" (UID: \"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.583147 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.583132 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.583288 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.583273 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.583592 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.583573 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc9e5acc-1a2a-4dd5-8cc5-800592c516bf-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-ggn8d\" (UID: \"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.601800 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.601778 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rngfm\" (UniqueName: \"kubernetes.io/projected/fc9e5acc-1a2a-4dd5-8cc5-800592c516bf-kube-api-access-rngfm\") pod \"istiod-openshift-gateway-94bb8fbfd-ggn8d\" (UID: \"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.601962 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.601939 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl8d5\" (UniqueName: \"kubernetes.io/projected/40fd4579-0c69-4093-a10d-438cb4f46f09-kube-api-access-nl8d5\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.603139 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.603116 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.605028 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.605003 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc9e5acc-1a2a-4dd5-8cc5-800592c516bf-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-ggn8d\" (UID: \"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.687869 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.687773 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:00.743180 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.743149 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:00.762967 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.762933 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:24:00.835149 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.834881 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw"] Jul 22 09:24:00.837661 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.837633 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:24:00.838220 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:24:00.838189 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40fd4579_0c69_4093_a10d_438cb4f46f09.slice/crio-ad9537a7f9836cd4c0a08633404fe8710fce04e02c1f373609711af62b122e72 WatchSource:0}: Error finding container ad9537a7f9836cd4c0a08633404fe8710fce04e02c1f373609711af62b122e72: Status 404 returned error can't find the container with id ad9537a7f9836cd4c0a08633404fe8710fce04e02c1f373609711af62b122e72 Jul 22 09:24:00.884362 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:00.884336 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d"] Jul 22 09:24:00.886718 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:24:00.886688 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9e5acc_1a2a_4dd5_8cc5_800592c516bf.slice/crio-72258328f9ffb96c1d3f7f2d3414e9e99bbdb79239f9612065f4f758d1308e4d WatchSource:0}: Error finding container 72258328f9ffb96c1d3f7f2d3414e9e99bbdb79239f9612065f4f758d1308e4d: Status 404 returned error can't find the container with id 72258328f9ffb96c1d3f7f2d3414e9e99bbdb79239f9612065f4f758d1308e4d Jul 22 09:24:01.839363 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:01.839266 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" event={"ID":"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf","Type":"ContainerStarted","Data":"72258328f9ffb96c1d3f7f2d3414e9e99bbdb79239f9612065f4f758d1308e4d"} Jul 22 09:24:01.840933 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:01.840906 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" event={"ID":"40fd4579-0c69-4093-a10d-438cb4f46f09","Type":"ContainerStarted","Data":"ad9537a7f9836cd4c0a08633404fe8710fce04e02c1f373609711af62b122e72"} Jul 22 09:24:03.536470 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:03.536431 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:24:03.626003 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:03.625966 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:24:03.720764 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:03.720693 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:24:04.106021 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:04.105983 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:24:04.213583 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:04.213537 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:24:04.585721 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:04.585683 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:24:04.585995 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:04.585749 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:24:04.590365 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:04.590331 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:24:04.590433 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:04.590403 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:24:04.590477 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:04.590436 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:24:04.857416 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:04.857068 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" event={"ID":"40fd4579-0c69-4093-a10d-438cb4f46f09","Type":"ContainerStarted","Data":"25823e1167faae044d33015a47f84e6c7f54cdff15503360121c3c833c8378f2"} Jul 22 09:24:04.858754 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:04.858728 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" event={"ID":"fc9e5acc-1a2a-4dd5-8cc5-800592c516bf","Type":"ContainerStarted","Data":"8ceff31e5fdaf967a1ba8d09910de9a153e4c2d60f299740efba839d29195121"} Jul 22 09:24:04.858899 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:04.858864 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:04.888687 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:04.888635 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" podStartSLOduration=1.13885359 podStartE2EDuration="4.888618664s" totalImagesPullingTime="3.749765074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:24:04.887104463 +0000 UTC m=+375.350898902" watchObservedRunningTime="2026-07-22 09:24:04.888618664 +0000 UTC m=+375.352413165" Jul 22 09:24:04.899923 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:04.899879 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:24:04.916449 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:04.916395 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" podStartSLOduration=1.21959221 podStartE2EDuration="4.916376937s" totalImagesPullingTime="3.696784727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:24:04.915266829 +0000 UTC m=+375.379061269" watchObservedRunningTime="2026-07-22 09:24:04.916376937 +0000 UTC m=+375.380171376" Jul 22 09:24:04.977758 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:04.977722 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:24:05.585570 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:05.585535 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw"] Jul 22 09:24:05.688138 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:05.688110 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:05.689486 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:05.689457 2549 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Jul 22 09:24:05.689586 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:05.689515 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" podUID="40fd4579-0c69-4093-a10d-438cb4f46f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Jul 22 09:24:05.870125 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:05.870044 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ggn8d" Jul 22 09:24:06.689034 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:06.688994 2549 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Jul 22 09:24:06.689539 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:06.689073 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" podUID="40fd4579-0c69-4093-a10d-438cb4f46f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Jul 22 09:24:06.872880 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:06.872844 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" podUID="40fd4579-0c69-4093-a10d-438cb4f46f09" containerName="istio-proxy" containerID="cri-o://25823e1167faae044d33015a47f84e6c7f54cdff15503360121c3c833c8378f2" gracePeriod=30 Jul 22 09:24:12.116211 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.116179 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:12.182817 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.182728 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-token\") pod \"40fd4579-0c69-4093-a10d-438cb4f46f09\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " Jul 22 09:24:12.182817 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.182770 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-envoy\") pod \"40fd4579-0c69-4093-a10d-438cb4f46f09\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " Jul 22 09:24:12.182817 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.182807 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40fd4579-0c69-4093-a10d-438cb4f46f09-istiod-ca-cert\") pod \"40fd4579-0c69-4093-a10d-438cb4f46f09\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " Jul 22 09:24:12.183072 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.182849 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-workload-certs\") pod \"40fd4579-0c69-4093-a10d-438cb4f46f09\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " Jul 22 09:24:12.183072 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.182878 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-workload-socket\") pod \"40fd4579-0c69-4093-a10d-438cb4f46f09\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " Jul 22 09:24:12.183072 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.182903 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-credential-socket\") pod \"40fd4579-0c69-4093-a10d-438cb4f46f09\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " Jul 22 09:24:12.183072 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.182946 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-data\") pod \"40fd4579-0c69-4093-a10d-438cb4f46f09\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " Jul 22 09:24:12.183072 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.182999 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-podinfo\") pod \"40fd4579-0c69-4093-a10d-438cb4f46f09\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " Jul 22 09:24:12.183072 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.183036 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nl8d5\" (UniqueName: \"kubernetes.io/projected/40fd4579-0c69-4093-a10d-438cb4f46f09-kube-api-access-nl8d5\") pod \"40fd4579-0c69-4093-a10d-438cb4f46f09\" (UID: \"40fd4579-0c69-4093-a10d-438cb4f46f09\") " Jul 22 09:24:12.183456 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.183180 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "40fd4579-0c69-4093-a10d-438cb4f46f09" (UID: "40fd4579-0c69-4093-a10d-438cb4f46f09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:24:12.183456 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.183245 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "40fd4579-0c69-4093-a10d-438cb4f46f09" (UID: "40fd4579-0c69-4093-a10d-438cb4f46f09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:24:12.183456 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.183265 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "40fd4579-0c69-4093-a10d-438cb4f46f09" (UID: "40fd4579-0c69-4093-a10d-438cb4f46f09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:24:12.183456 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.183286 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40fd4579-0c69-4093-a10d-438cb4f46f09-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "40fd4579-0c69-4093-a10d-438cb4f46f09" (UID: "40fd4579-0c69-4093-a10d-438cb4f46f09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:24:12.183619 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.183459 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-data" (OuterVolumeSpecName: "istio-data") pod "40fd4579-0c69-4093-a10d-438cb4f46f09" (UID: "40fd4579-0c69-4093-a10d-438cb4f46f09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:24:12.185106 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.185078 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "40fd4579-0c69-4093-a10d-438cb4f46f09" (UID: "40fd4579-0c69-4093-a10d-438cb4f46f09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:24:12.185203 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.185100 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-token" (OuterVolumeSpecName: "istio-token") pod "40fd4579-0c69-4093-a10d-438cb4f46f09" (UID: "40fd4579-0c69-4093-a10d-438cb4f46f09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:24:12.185243 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.185200 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "40fd4579-0c69-4093-a10d-438cb4f46f09" (UID: "40fd4579-0c69-4093-a10d-438cb4f46f09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 09:24:12.185243 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.185225 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40fd4579-0c69-4093-a10d-438cb4f46f09-kube-api-access-nl8d5" (OuterVolumeSpecName: "kube-api-access-nl8d5") pod "40fd4579-0c69-4093-a10d-438cb4f46f09" (UID: "40fd4579-0c69-4093-a10d-438cb4f46f09"). InnerVolumeSpecName "kube-api-access-nl8d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:24:12.284634 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.284595 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:24:12.284634 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.284625 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:24:12.284634 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.284635 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:24:12.284634 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.284644 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:24:12.284880 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.284653 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl8d5\" (UniqueName: \"kubernetes.io/projected/40fd4579-0c69-4093-a10d-438cb4f46f09-kube-api-access-nl8d5\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:24:12.284880 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.284661 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:24:12.284880 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.284669 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:24:12.284880 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.284676 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40fd4579-0c69-4093-a10d-438cb4f46f09-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:24:12.284880 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.284683 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40fd4579-0c69-4093-a10d-438cb4f46f09-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:24:12.898258 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.898216 2549 generic.go:356] "Generic (PLEG): container finished" podID="40fd4579-0c69-4093-a10d-438cb4f46f09" containerID="25823e1167faae044d33015a47f84e6c7f54cdff15503360121c3c833c8378f2" exitCode=0 Jul 22 09:24:12.898449 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.898268 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" Jul 22 09:24:12.898449 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.898267 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" event={"ID":"40fd4579-0c69-4093-a10d-438cb4f46f09","Type":"ContainerDied","Data":"25823e1167faae044d33015a47f84e6c7f54cdff15503360121c3c833c8378f2"} Jul 22 09:24:12.898449 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.898384 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw" event={"ID":"40fd4579-0c69-4093-a10d-438cb4f46f09","Type":"ContainerDied","Data":"ad9537a7f9836cd4c0a08633404fe8710fce04e02c1f373609711af62b122e72"} Jul 22 09:24:12.898449 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.898412 2549 scope.go:117] "RemoveContainer" containerID="25823e1167faae044d33015a47f84e6c7f54cdff15503360121c3c833c8378f2" Jul 22 09:24:12.906413 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.906390 2549 scope.go:117] "RemoveContainer" containerID="25823e1167faae044d33015a47f84e6c7f54cdff15503360121c3c833c8378f2" Jul 22 09:24:12.906656 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:24:12.906638 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25823e1167faae044d33015a47f84e6c7f54cdff15503360121c3c833c8378f2\": container with ID starting with 25823e1167faae044d33015a47f84e6c7f54cdff15503360121c3c833c8378f2 not found: ID does not exist" containerID="25823e1167faae044d33015a47f84e6c7f54cdff15503360121c3c833c8378f2" Jul 22 09:24:12.906701 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.906664 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25823e1167faae044d33015a47f84e6c7f54cdff15503360121c3c833c8378f2"} err="failed to get container status \"25823e1167faae044d33015a47f84e6c7f54cdff15503360121c3c833c8378f2\": rpc error: code = NotFound desc = could not find container \"25823e1167faae044d33015a47f84e6c7f54cdff15503360121c3c833c8378f2\": container with ID starting with 25823e1167faae044d33015a47f84e6c7f54cdff15503360121c3c833c8378f2 not found: ID does not exist" Jul 22 09:24:12.918873 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.918844 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw"] Jul 22 09:24:12.924743 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:12.924719 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-fl5zw"] Jul 22 09:24:14.338998 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:14.338965 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40fd4579-0c69-4093-a10d-438cb4f46f09" path="/var/lib/kubelet/pods/40fd4579-0c69-4093-a10d-438cb4f46f09/volumes" Jul 22 09:24:36.613849 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:36.613810 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:24:36.675530 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:36.675493 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:24:38.459384 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:38.459349 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:24:38.560887 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:38.560840 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:24:52.730922 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:52.730882 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:24:52.829575 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:52.829542 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:24:52.969511 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:52.969474 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:24:59.400706 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:59.400659 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:24:59.502669 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:24:59.502634 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:08.041554 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.041516 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb"] Jul 22 09:25:08.041936 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.041795 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40fd4579-0c69-4093-a10d-438cb4f46f09" containerName="istio-proxy" Jul 22 09:25:08.041936 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.041806 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="40fd4579-0c69-4093-a10d-438cb4f46f09" containerName="istio-proxy" Jul 22 09:25:08.041936 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.041874 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="40fd4579-0c69-4093-a10d-438cb4f46f09" containerName="istio-proxy" Jul 22 09:25:08.046038 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.046020 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb" Jul 22 09:25:08.048711 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.048693 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 09:25:08.049688 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.049659 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wnw8l\"" Jul 22 09:25:08.049803 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.049652 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 09:25:08.061389 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.061362 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb"] Jul 22 09:25:08.133163 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.133128 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f2bc8a1-a99e-4238-9f60-24098f97e5d9-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb\" (UID: \"5f2bc8a1-a99e-4238-9f60-24098f97e5d9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb" Jul 22 09:25:08.133163 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.133162 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f2bc8a1-a99e-4238-9f60-24098f97e5d9-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb\" (UID: \"5f2bc8a1-a99e-4238-9f60-24098f97e5d9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb" Jul 22 09:25:08.133395 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.133204 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vthsz\" (UniqueName: \"kubernetes.io/projected/5f2bc8a1-a99e-4238-9f60-24098f97e5d9-kube-api-access-vthsz\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb\" (UID: \"5f2bc8a1-a99e-4238-9f60-24098f97e5d9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb" Jul 22 09:25:08.234601 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.234566 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f2bc8a1-a99e-4238-9f60-24098f97e5d9-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb\" (UID: \"5f2bc8a1-a99e-4238-9f60-24098f97e5d9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb" Jul 22 09:25:08.234601 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.234603 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f2bc8a1-a99e-4238-9f60-24098f97e5d9-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb\" (UID: \"5f2bc8a1-a99e-4238-9f60-24098f97e5d9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb" Jul 22 09:25:08.234837 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.234652 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vthsz\" (UniqueName: \"kubernetes.io/projected/5f2bc8a1-a99e-4238-9f60-24098f97e5d9-kube-api-access-vthsz\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb\" (UID: \"5f2bc8a1-a99e-4238-9f60-24098f97e5d9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb" Jul 22 09:25:08.234950 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.234929 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f2bc8a1-a99e-4238-9f60-24098f97e5d9-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb\" (UID: \"5f2bc8a1-a99e-4238-9f60-24098f97e5d9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb" Jul 22 09:25:08.235032 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.235013 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f2bc8a1-a99e-4238-9f60-24098f97e5d9-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb\" (UID: \"5f2bc8a1-a99e-4238-9f60-24098f97e5d9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb" Jul 22 09:25:08.244131 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.244106 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vthsz\" (UniqueName: \"kubernetes.io/projected/5f2bc8a1-a99e-4238-9f60-24098f97e5d9-kube-api-access-vthsz\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb\" (UID: \"5f2bc8a1-a99e-4238-9f60-24098f97e5d9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb" Jul 22 09:25:08.355264 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.355177 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb" Jul 22 09:25:08.501383 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.501356 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb"] Jul 22 09:25:08.503856 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:25:08.503826 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2bc8a1_a99e_4238_9f60_24098f97e5d9.slice/crio-6c518869fb79958946e6f2aac681b0c8bc4da40651b67d4fa4a05de7ca03d824 WatchSource:0}: Error finding container 6c518869fb79958946e6f2aac681b0c8bc4da40651b67d4fa4a05de7ca03d824: Status 404 returned error can't find the container with id 6c518869fb79958946e6f2aac681b0c8bc4da40651b67d4fa4a05de7ca03d824 Jul 22 09:25:08.672126 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.672089 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb"] Jul 22 09:25:08.675410 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.675395 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb" Jul 22 09:25:08.699986 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.699956 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb"] Jul 22 09:25:08.738762 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.738727 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/481aaf1f-a26a-41d3-b266-e439c874e042-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb\" (UID: \"481aaf1f-a26a-41d3-b266-e439c874e042\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb" Jul 22 09:25:08.738951 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.738781 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/481aaf1f-a26a-41d3-b266-e439c874e042-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb\" (UID: \"481aaf1f-a26a-41d3-b266-e439c874e042\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb" Jul 22 09:25:08.738951 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.738825 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sqvg\" (UniqueName: \"kubernetes.io/projected/481aaf1f-a26a-41d3-b266-e439c874e042-kube-api-access-7sqvg\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb\" (UID: \"481aaf1f-a26a-41d3-b266-e439c874e042\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb" Jul 22 09:25:08.839250 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.839211 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/481aaf1f-a26a-41d3-b266-e439c874e042-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb\" (UID: \"481aaf1f-a26a-41d3-b266-e439c874e042\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb" Jul 22 09:25:08.839250 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.839253 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sqvg\" (UniqueName: \"kubernetes.io/projected/481aaf1f-a26a-41d3-b266-e439c874e042-kube-api-access-7sqvg\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb\" (UID: \"481aaf1f-a26a-41d3-b266-e439c874e042\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb" Jul 22 09:25:08.839477 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.839321 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/481aaf1f-a26a-41d3-b266-e439c874e042-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb\" (UID: \"481aaf1f-a26a-41d3-b266-e439c874e042\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb" Jul 22 09:25:08.839640 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.839622 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/481aaf1f-a26a-41d3-b266-e439c874e042-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb\" (UID: \"481aaf1f-a26a-41d3-b266-e439c874e042\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb" Jul 22 09:25:08.839677 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.839648 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/481aaf1f-a26a-41d3-b266-e439c874e042-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb\" (UID: \"481aaf1f-a26a-41d3-b266-e439c874e042\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb" Jul 22 09:25:08.853930 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.853901 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sqvg\" (UniqueName: \"kubernetes.io/projected/481aaf1f-a26a-41d3-b266-e439c874e042-kube-api-access-7sqvg\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb\" (UID: \"481aaf1f-a26a-41d3-b266-e439c874e042\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb" Jul 22 09:25:08.983827 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:08.983746 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb" Jul 22 09:25:09.100108 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.100073 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh"] Jul 22 09:25:09.104583 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.104561 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh" Jul 22 09:25:09.140017 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.139991 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb"] Jul 22 09:25:09.141648 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:25:09.141622 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481aaf1f_a26a_41d3_b266_e439c874e042.slice/crio-9c0d93de247dbeefb8a9bb49fb76a5235310e77756f111276b5385e32fa26398 WatchSource:0}: Error finding container 9c0d93de247dbeefb8a9bb49fb76a5235310e77756f111276b5385e32fa26398: Status 404 returned error can't find the container with id 9c0d93de247dbeefb8a9bb49fb76a5235310e77756f111276b5385e32fa26398 Jul 22 09:25:09.145362 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.145332 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh"] Jul 22 09:25:09.145975 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.145953 2549 generic.go:356] "Generic (PLEG): container finished" podID="5f2bc8a1-a99e-4238-9f60-24098f97e5d9" containerID="df6f30a77b78b11fe14d754f4fd8d1c4705573a811c18b987a48aafc8aedd420" exitCode=0 Jul 22 09:25:09.146041 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.146015 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb" event={"ID":"5f2bc8a1-a99e-4238-9f60-24098f97e5d9","Type":"ContainerDied","Data":"df6f30a77b78b11fe14d754f4fd8d1c4705573a811c18b987a48aafc8aedd420"} Jul 22 09:25:09.146079 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.146048 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb" event={"ID":"5f2bc8a1-a99e-4238-9f60-24098f97e5d9","Type":"ContainerStarted","Data":"6c518869fb79958946e6f2aac681b0c8bc4da40651b67d4fa4a05de7ca03d824"} Jul 22 09:25:09.241402 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.241362 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e832e7ea-798d-488b-99e7-729ae596b1db-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh\" (UID: \"e832e7ea-798d-488b-99e7-729ae596b1db\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh" Jul 22 09:25:09.241536 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.241429 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e832e7ea-798d-488b-99e7-729ae596b1db-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh\" (UID: \"e832e7ea-798d-488b-99e7-729ae596b1db\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh" Jul 22 09:25:09.241536 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.241511 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8zsj\" (UniqueName: \"kubernetes.io/projected/e832e7ea-798d-488b-99e7-729ae596b1db-kube-api-access-t8zsj\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh\" (UID: \"e832e7ea-798d-488b-99e7-729ae596b1db\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh" Jul 22 09:25:09.341936 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.341897 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e832e7ea-798d-488b-99e7-729ae596b1db-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh\" (UID: \"e832e7ea-798d-488b-99e7-729ae596b1db\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh" Jul 22 09:25:09.342201 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.341950 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e832e7ea-798d-488b-99e7-729ae596b1db-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh\" (UID: \"e832e7ea-798d-488b-99e7-729ae596b1db\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh" Jul 22 09:25:09.342201 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.342065 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8zsj\" (UniqueName: \"kubernetes.io/projected/e832e7ea-798d-488b-99e7-729ae596b1db-kube-api-access-t8zsj\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh\" (UID: \"e832e7ea-798d-488b-99e7-729ae596b1db\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh" Jul 22 09:25:09.342283 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.342258 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e832e7ea-798d-488b-99e7-729ae596b1db-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh\" (UID: \"e832e7ea-798d-488b-99e7-729ae596b1db\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh" Jul 22 09:25:09.342352 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.342318 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e832e7ea-798d-488b-99e7-729ae596b1db-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh\" (UID: \"e832e7ea-798d-488b-99e7-729ae596b1db\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh" Jul 22 09:25:09.353176 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.353148 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8zsj\" (UniqueName: \"kubernetes.io/projected/e832e7ea-798d-488b-99e7-729ae596b1db-kube-api-access-t8zsj\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh\" (UID: \"e832e7ea-798d-488b-99e7-729ae596b1db\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh" Jul 22 09:25:09.413153 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.413116 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh" Jul 22 09:25:09.475427 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.475398 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k"] Jul 22 09:25:09.479143 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.479123 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k" Jul 22 09:25:09.501227 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.501194 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k"] Jul 22 09:25:09.544729 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.544695 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f7434ca-9a9b-40d4-8219-d03a2e4a05f1-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k\" (UID: \"7f7434ca-9a9b-40d4-8219-d03a2e4a05f1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k" Jul 22 09:25:09.544729 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.544730 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f7434ca-9a9b-40d4-8219-d03a2e4a05f1-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k\" (UID: \"7f7434ca-9a9b-40d4-8219-d03a2e4a05f1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k" Jul 22 09:25:09.544925 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.544801 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75zsd\" (UniqueName: \"kubernetes.io/projected/7f7434ca-9a9b-40d4-8219-d03a2e4a05f1-kube-api-access-75zsd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k\" (UID: \"7f7434ca-9a9b-40d4-8219-d03a2e4a05f1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k" Jul 22 09:25:09.559969 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.559859 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh"] Jul 22 09:25:09.562622 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:25:09.562593 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode832e7ea_798d_488b_99e7_729ae596b1db.slice/crio-c9f4415fbaf8664d4110e55d7a6d531110d180379f3d2dcd2adbbd5eda8ffb32 WatchSource:0}: Error finding container c9f4415fbaf8664d4110e55d7a6d531110d180379f3d2dcd2adbbd5eda8ffb32: Status 404 returned error can't find the container with id c9f4415fbaf8664d4110e55d7a6d531110d180379f3d2dcd2adbbd5eda8ffb32 Jul 22 09:25:09.645916 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.645869 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75zsd\" (UniqueName: \"kubernetes.io/projected/7f7434ca-9a9b-40d4-8219-d03a2e4a05f1-kube-api-access-75zsd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k\" (UID: \"7f7434ca-9a9b-40d4-8219-d03a2e4a05f1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k" Jul 22 09:25:09.645916 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.645921 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f7434ca-9a9b-40d4-8219-d03a2e4a05f1-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k\" (UID: \"7f7434ca-9a9b-40d4-8219-d03a2e4a05f1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k" Jul 22 09:25:09.646186 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.645941 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f7434ca-9a9b-40d4-8219-d03a2e4a05f1-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k\" (UID: \"7f7434ca-9a9b-40d4-8219-d03a2e4a05f1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k" Jul 22 09:25:09.646386 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.646361 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f7434ca-9a9b-40d4-8219-d03a2e4a05f1-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k\" (UID: \"7f7434ca-9a9b-40d4-8219-d03a2e4a05f1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k" Jul 22 09:25:09.646465 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.646371 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f7434ca-9a9b-40d4-8219-d03a2e4a05f1-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k\" (UID: \"7f7434ca-9a9b-40d4-8219-d03a2e4a05f1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k" Jul 22 09:25:09.658393 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.658370 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75zsd\" (UniqueName: \"kubernetes.io/projected/7f7434ca-9a9b-40d4-8219-d03a2e4a05f1-kube-api-access-75zsd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k\" (UID: \"7f7434ca-9a9b-40d4-8219-d03a2e4a05f1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k" Jul 22 09:25:09.789144 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.789118 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k" Jul 22 09:25:09.930220 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:09.930193 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k"] Jul 22 09:25:09.934112 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:25:09.934088 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7434ca_9a9b_40d4_8219_d03a2e4a05f1.slice/crio-4161d66d29b3dd42079be8defeece940517f792b9ee0ceab0f14aaf455d35f79 WatchSource:0}: Error finding container 4161d66d29b3dd42079be8defeece940517f792b9ee0ceab0f14aaf455d35f79: Status 404 returned error can't find the container with id 4161d66d29b3dd42079be8defeece940517f792b9ee0ceab0f14aaf455d35f79 Jul 22 09:25:10.152551 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:10.152520 2549 generic.go:356] "Generic (PLEG): container finished" podID="5f2bc8a1-a99e-4238-9f60-24098f97e5d9" containerID="9ddd288126187ad1ec347e7496c6e1bdbd3180fd9391cb62bda0b8e53e70c07f" exitCode=0 Jul 22 09:25:10.152992 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:10.152611 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb" event={"ID":"5f2bc8a1-a99e-4238-9f60-24098f97e5d9","Type":"ContainerDied","Data":"9ddd288126187ad1ec347e7496c6e1bdbd3180fd9391cb62bda0b8e53e70c07f"} Jul 22 09:25:10.154160 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:10.154140 2549 generic.go:356] "Generic (PLEG): container finished" podID="481aaf1f-a26a-41d3-b266-e439c874e042" containerID="edf34cc9f2eada0605f88dc1e5ee3f1fee6eb8a0ae4964e5bcd7bb0d867703a9" exitCode=0 Jul 22 09:25:10.154255 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:10.154217 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb" event={"ID":"481aaf1f-a26a-41d3-b266-e439c874e042","Type":"ContainerDied","Data":"edf34cc9f2eada0605f88dc1e5ee3f1fee6eb8a0ae4964e5bcd7bb0d867703a9"} Jul 22 09:25:10.154255 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:10.154252 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb" event={"ID":"481aaf1f-a26a-41d3-b266-e439c874e042","Type":"ContainerStarted","Data":"9c0d93de247dbeefb8a9bb49fb76a5235310e77756f111276b5385e32fa26398"} Jul 22 09:25:10.155730 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:10.155710 2549 generic.go:356] "Generic (PLEG): container finished" podID="e832e7ea-798d-488b-99e7-729ae596b1db" containerID="70c5542f7e2507f57f53eed4a8d96dcda5f48261cea7a289dd9bb9174e36461e" exitCode=0 Jul 22 09:25:10.155971 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:10.155802 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh" event={"ID":"e832e7ea-798d-488b-99e7-729ae596b1db","Type":"ContainerDied","Data":"70c5542f7e2507f57f53eed4a8d96dcda5f48261cea7a289dd9bb9174e36461e"} Jul 22 09:25:10.155971 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:10.155845 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh" event={"ID":"e832e7ea-798d-488b-99e7-729ae596b1db","Type":"ContainerStarted","Data":"c9f4415fbaf8664d4110e55d7a6d531110d180379f3d2dcd2adbbd5eda8ffb32"} Jul 22 09:25:10.157418 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:10.157388 2549 generic.go:356] "Generic (PLEG): container finished" podID="7f7434ca-9a9b-40d4-8219-d03a2e4a05f1" containerID="929310c01914fc64609e14498c659de6730407dc04277f1acb58c4ebf09f0b89" exitCode=0 Jul 22 09:25:10.157499 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:10.157428 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k" event={"ID":"7f7434ca-9a9b-40d4-8219-d03a2e4a05f1","Type":"ContainerDied","Data":"929310c01914fc64609e14498c659de6730407dc04277f1acb58c4ebf09f0b89"} Jul 22 09:25:10.157499 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:10.157455 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k" event={"ID":"7f7434ca-9a9b-40d4-8219-d03a2e4a05f1","Type":"ContainerStarted","Data":"4161d66d29b3dd42079be8defeece940517f792b9ee0ceab0f14aaf455d35f79"} Jul 22 09:25:11.164573 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:11.164512 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh" event={"ID":"e832e7ea-798d-488b-99e7-729ae596b1db","Type":"ContainerStarted","Data":"3e92da159fc15abc6846d440d9cc83589c34ed09a4166ef603389bcb75c09bc4"} Jul 22 09:25:11.166952 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:11.166648 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k" event={"ID":"7f7434ca-9a9b-40d4-8219-d03a2e4a05f1","Type":"ContainerStarted","Data":"1070deafdd74f4ef074090dadb5ccf65a2100e4ca038ba475224277ca8a113a2"} Jul 22 09:25:11.169183 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:11.169155 2549 generic.go:356] "Generic (PLEG): container finished" podID="5f2bc8a1-a99e-4238-9f60-24098f97e5d9" containerID="cc2948893fd8a4d136af3a2fef41ea6e85d5e19e968280a7a0614faac4c490b6" exitCode=0 Jul 22 09:25:11.169267 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:11.169211 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb" event={"ID":"5f2bc8a1-a99e-4238-9f60-24098f97e5d9","Type":"ContainerDied","Data":"cc2948893fd8a4d136af3a2fef41ea6e85d5e19e968280a7a0614faac4c490b6"} Jul 22 09:25:12.175162 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:12.175077 2549 generic.go:356] "Generic (PLEG): container finished" podID="7f7434ca-9a9b-40d4-8219-d03a2e4a05f1" containerID="1070deafdd74f4ef074090dadb5ccf65a2100e4ca038ba475224277ca8a113a2" exitCode=0 Jul 22 09:25:12.175583 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:12.175168 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k" event={"ID":"7f7434ca-9a9b-40d4-8219-d03a2e4a05f1","Type":"ContainerDied","Data":"1070deafdd74f4ef074090dadb5ccf65a2100e4ca038ba475224277ca8a113a2"} Jul 22 09:25:12.177099 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:12.177080 2549 generic.go:356] "Generic (PLEG): container finished" podID="481aaf1f-a26a-41d3-b266-e439c874e042" containerID="a9c84019facf020bf90bdb042ae7a0d78d944a1bb298d1d9bbb4d343902d6858" exitCode=0 Jul 22 09:25:12.177186 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:12.177105 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb" event={"ID":"481aaf1f-a26a-41d3-b266-e439c874e042","Type":"ContainerDied","Data":"a9c84019facf020bf90bdb042ae7a0d78d944a1bb298d1d9bbb4d343902d6858"} Jul 22 09:25:12.178910 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:12.178891 2549 generic.go:356] "Generic (PLEG): container finished" podID="e832e7ea-798d-488b-99e7-729ae596b1db" containerID="3e92da159fc15abc6846d440d9cc83589c34ed09a4166ef603389bcb75c09bc4" exitCode=0 Jul 22 09:25:12.178997 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:12.178967 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh" event={"ID":"e832e7ea-798d-488b-99e7-729ae596b1db","Type":"ContainerDied","Data":"3e92da159fc15abc6846d440d9cc83589c34ed09a4166ef603389bcb75c09bc4"} Jul 22 09:25:12.326973 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:12.326945 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb" Jul 22 09:25:12.466260 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:12.466235 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f2bc8a1-a99e-4238-9f60-24098f97e5d9-util\") pod \"5f2bc8a1-a99e-4238-9f60-24098f97e5d9\" (UID: \"5f2bc8a1-a99e-4238-9f60-24098f97e5d9\") " Jul 22 09:25:12.466380 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:12.466279 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vthsz\" (UniqueName: \"kubernetes.io/projected/5f2bc8a1-a99e-4238-9f60-24098f97e5d9-kube-api-access-vthsz\") pod \"5f2bc8a1-a99e-4238-9f60-24098f97e5d9\" (UID: \"5f2bc8a1-a99e-4238-9f60-24098f97e5d9\") " Jul 22 09:25:12.466380 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:12.466361 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f2bc8a1-a99e-4238-9f60-24098f97e5d9-bundle\") pod \"5f2bc8a1-a99e-4238-9f60-24098f97e5d9\" (UID: \"5f2bc8a1-a99e-4238-9f60-24098f97e5d9\") " Jul 22 09:25:12.466985 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:12.466953 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2bc8a1-a99e-4238-9f60-24098f97e5d9-bundle" (OuterVolumeSpecName: "bundle") pod "5f2bc8a1-a99e-4238-9f60-24098f97e5d9" (UID: "5f2bc8a1-a99e-4238-9f60-24098f97e5d9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:25:12.468437 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:12.468416 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2bc8a1-a99e-4238-9f60-24098f97e5d9-kube-api-access-vthsz" (OuterVolumeSpecName: "kube-api-access-vthsz") pod "5f2bc8a1-a99e-4238-9f60-24098f97e5d9" (UID: "5f2bc8a1-a99e-4238-9f60-24098f97e5d9"). InnerVolumeSpecName "kube-api-access-vthsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:25:12.471763 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:12.471742 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2bc8a1-a99e-4238-9f60-24098f97e5d9-util" (OuterVolumeSpecName: "util") pod "5f2bc8a1-a99e-4238-9f60-24098f97e5d9" (UID: "5f2bc8a1-a99e-4238-9f60-24098f97e5d9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:25:12.567645 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:12.567610 2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f2bc8a1-a99e-4238-9f60-24098f97e5d9-bundle\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:25:12.567645 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:12.567643 2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f2bc8a1-a99e-4238-9f60-24098f97e5d9-util\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:25:12.567645 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:12.567653 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vthsz\" (UniqueName: \"kubernetes.io/projected/5f2bc8a1-a99e-4238-9f60-24098f97e5d9-kube-api-access-vthsz\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:25:13.186354 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:13.186319 2549 generic.go:356] "Generic (PLEG): container finished" podID="7f7434ca-9a9b-40d4-8219-d03a2e4a05f1" containerID="f9ccca34235f3d1be8b7c66ca03025b311ca3a9c313838cbe2a2d54b1896a3bc" exitCode=0 Jul 22 09:25:13.186789 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:13.186397 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k" event={"ID":"7f7434ca-9a9b-40d4-8219-d03a2e4a05f1","Type":"ContainerDied","Data":"f9ccca34235f3d1be8b7c66ca03025b311ca3a9c313838cbe2a2d54b1896a3bc"} Jul 22 09:25:13.188111 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:13.188092 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb" Jul 22 09:25:13.188111 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:13.188096 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb" event={"ID":"5f2bc8a1-a99e-4238-9f60-24098f97e5d9","Type":"ContainerDied","Data":"6c518869fb79958946e6f2aac681b0c8bc4da40651b67d4fa4a05de7ca03d824"} Jul 22 09:25:13.188265 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:13.188132 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c518869fb79958946e6f2aac681b0c8bc4da40651b67d4fa4a05de7ca03d824" Jul 22 09:25:13.189964 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:13.189942 2549 generic.go:356] "Generic (PLEG): container finished" podID="481aaf1f-a26a-41d3-b266-e439c874e042" containerID="51526800d41e8db964b402a120bbcaf4c74be9a2b35ba07ad28035ccfa4f8d0f" exitCode=0 Jul 22 09:25:13.190069 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:13.190016 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb" event={"ID":"481aaf1f-a26a-41d3-b266-e439c874e042","Type":"ContainerDied","Data":"51526800d41e8db964b402a120bbcaf4c74be9a2b35ba07ad28035ccfa4f8d0f"} Jul 22 09:25:13.192032 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:13.192010 2549 generic.go:356] "Generic (PLEG): container finished" podID="e832e7ea-798d-488b-99e7-729ae596b1db" containerID="ed56842cbf63c0337dff20591853a3bad42cbbff2bfe29a3a48328cb1f9911c0" exitCode=0 Jul 22 09:25:13.192130 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:13.192045 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh" event={"ID":"e832e7ea-798d-488b-99e7-729ae596b1db","Type":"ContainerDied","Data":"ed56842cbf63c0337dff20591853a3bad42cbbff2bfe29a3a48328cb1f9911c0"} Jul 22 09:25:14.342452 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.342427 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh" Jul 22 09:25:14.382680 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.382429 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb" Jul 22 09:25:14.384083 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.384058 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k" Jul 22 09:25:14.483843 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.483774 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sqvg\" (UniqueName: \"kubernetes.io/projected/481aaf1f-a26a-41d3-b266-e439c874e042-kube-api-access-7sqvg\") pod \"481aaf1f-a26a-41d3-b266-e439c874e042\" (UID: \"481aaf1f-a26a-41d3-b266-e439c874e042\") " Jul 22 09:25:14.483843 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.483816 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8zsj\" (UniqueName: \"kubernetes.io/projected/e832e7ea-798d-488b-99e7-729ae596b1db-kube-api-access-t8zsj\") pod \"e832e7ea-798d-488b-99e7-729ae596b1db\" (UID: \"e832e7ea-798d-488b-99e7-729ae596b1db\") " Jul 22 09:25:14.484061 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.483844 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/481aaf1f-a26a-41d3-b266-e439c874e042-util\") pod \"481aaf1f-a26a-41d3-b266-e439c874e042\" (UID: \"481aaf1f-a26a-41d3-b266-e439c874e042\") " Jul 22 09:25:14.484061 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.483866 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e832e7ea-798d-488b-99e7-729ae596b1db-util\") pod \"e832e7ea-798d-488b-99e7-729ae596b1db\" (UID: \"e832e7ea-798d-488b-99e7-729ae596b1db\") " Jul 22 09:25:14.484061 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.483984 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f7434ca-9a9b-40d4-8219-d03a2e4a05f1-bundle\") pod \"7f7434ca-9a9b-40d4-8219-d03a2e4a05f1\" (UID: \"7f7434ca-9a9b-40d4-8219-d03a2e4a05f1\") " Jul 22 09:25:14.484061 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.484036 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f7434ca-9a9b-40d4-8219-d03a2e4a05f1-util\") pod \"7f7434ca-9a9b-40d4-8219-d03a2e4a05f1\" (UID: \"7f7434ca-9a9b-40d4-8219-d03a2e4a05f1\") " Jul 22 09:25:14.484262 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.484069 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75zsd\" (UniqueName: \"kubernetes.io/projected/7f7434ca-9a9b-40d4-8219-d03a2e4a05f1-kube-api-access-75zsd\") pod \"7f7434ca-9a9b-40d4-8219-d03a2e4a05f1\" (UID: \"7f7434ca-9a9b-40d4-8219-d03a2e4a05f1\") " Jul 22 09:25:14.484262 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.484114 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/481aaf1f-a26a-41d3-b266-e439c874e042-bundle\") pod \"481aaf1f-a26a-41d3-b266-e439c874e042\" (UID: \"481aaf1f-a26a-41d3-b266-e439c874e042\") " Jul 22 09:25:14.484262 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.484165 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e832e7ea-798d-488b-99e7-729ae596b1db-bundle\") pod \"e832e7ea-798d-488b-99e7-729ae596b1db\" (UID: \"e832e7ea-798d-488b-99e7-729ae596b1db\") " Jul 22 09:25:14.485548 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.484752 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e832e7ea-798d-488b-99e7-729ae596b1db-bundle" (OuterVolumeSpecName: "bundle") pod "e832e7ea-798d-488b-99e7-729ae596b1db" (UID: "e832e7ea-798d-488b-99e7-729ae596b1db"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:25:14.485548 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.484750 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f7434ca-9a9b-40d4-8219-d03a2e4a05f1-bundle" (OuterVolumeSpecName: "bundle") pod "7f7434ca-9a9b-40d4-8219-d03a2e4a05f1" (UID: "7f7434ca-9a9b-40d4-8219-d03a2e4a05f1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:25:14.486240 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.486194 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481aaf1f-a26a-41d3-b266-e439c874e042-bundle" (OuterVolumeSpecName: "bundle") pod "481aaf1f-a26a-41d3-b266-e439c874e042" (UID: "481aaf1f-a26a-41d3-b266-e439c874e042"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:25:14.486366 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.486336 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e832e7ea-798d-488b-99e7-729ae596b1db-kube-api-access-t8zsj" (OuterVolumeSpecName: "kube-api-access-t8zsj") pod "e832e7ea-798d-488b-99e7-729ae596b1db" (UID: "e832e7ea-798d-488b-99e7-729ae596b1db"). InnerVolumeSpecName "kube-api-access-t8zsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:25:14.486611 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.486581 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7434ca-9a9b-40d4-8219-d03a2e4a05f1-kube-api-access-75zsd" (OuterVolumeSpecName: "kube-api-access-75zsd") pod "7f7434ca-9a9b-40d4-8219-d03a2e4a05f1" (UID: "7f7434ca-9a9b-40d4-8219-d03a2e4a05f1"). InnerVolumeSpecName "kube-api-access-75zsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:25:14.486810 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.486791 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481aaf1f-a26a-41d3-b266-e439c874e042-kube-api-access-7sqvg" (OuterVolumeSpecName: "kube-api-access-7sqvg") pod "481aaf1f-a26a-41d3-b266-e439c874e042" (UID: "481aaf1f-a26a-41d3-b266-e439c874e042"). InnerVolumeSpecName "kube-api-access-7sqvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:25:14.490484 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.490446 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f7434ca-9a9b-40d4-8219-d03a2e4a05f1-util" (OuterVolumeSpecName: "util") pod "7f7434ca-9a9b-40d4-8219-d03a2e4a05f1" (UID: "7f7434ca-9a9b-40d4-8219-d03a2e4a05f1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:25:14.490805 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.490769 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e832e7ea-798d-488b-99e7-729ae596b1db-util" (OuterVolumeSpecName: "util") pod "e832e7ea-798d-488b-99e7-729ae596b1db" (UID: "e832e7ea-798d-488b-99e7-729ae596b1db"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:25:14.501403 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.501373 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481aaf1f-a26a-41d3-b266-e439c874e042-util" (OuterVolumeSpecName: "util") pod "481aaf1f-a26a-41d3-b266-e439c874e042" (UID: "481aaf1f-a26a-41d3-b266-e439c874e042"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:25:14.585566 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.585536 2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f7434ca-9a9b-40d4-8219-d03a2e4a05f1-util\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:25:14.585566 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.585565 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75zsd\" (UniqueName: \"kubernetes.io/projected/7f7434ca-9a9b-40d4-8219-d03a2e4a05f1-kube-api-access-75zsd\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:25:14.585730 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.585576 2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/481aaf1f-a26a-41d3-b266-e439c874e042-bundle\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:25:14.585730 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.585586 2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e832e7ea-798d-488b-99e7-729ae596b1db-bundle\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:25:14.585730 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.585594 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sqvg\" (UniqueName: \"kubernetes.io/projected/481aaf1f-a26a-41d3-b266-e439c874e042-kube-api-access-7sqvg\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:25:14.585730 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.585603 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8zsj\" (UniqueName: \"kubernetes.io/projected/e832e7ea-798d-488b-99e7-729ae596b1db-kube-api-access-t8zsj\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:25:14.585730 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.585611 2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/481aaf1f-a26a-41d3-b266-e439c874e042-util\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:25:14.585730 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.585619 2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e832e7ea-798d-488b-99e7-729ae596b1db-util\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:25:14.585730 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:14.585626 2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f7434ca-9a9b-40d4-8219-d03a2e4a05f1-bundle\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:25:15.205601 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:15.205561 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k" Jul 22 09:25:15.205771 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:15.205558 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k" event={"ID":"7f7434ca-9a9b-40d4-8219-d03a2e4a05f1","Type":"ContainerDied","Data":"4161d66d29b3dd42079be8defeece940517f792b9ee0ceab0f14aaf455d35f79"} Jul 22 09:25:15.205771 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:15.205689 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4161d66d29b3dd42079be8defeece940517f792b9ee0ceab0f14aaf455d35f79" Jul 22 09:25:15.207493 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:15.207466 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb" event={"ID":"481aaf1f-a26a-41d3-b266-e439c874e042","Type":"ContainerDied","Data":"9c0d93de247dbeefb8a9bb49fb76a5235310e77756f111276b5385e32fa26398"} Jul 22 09:25:15.207493 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:15.207482 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb" Jul 22 09:25:15.207694 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:15.207507 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c0d93de247dbeefb8a9bb49fb76a5235310e77756f111276b5385e32fa26398" Jul 22 09:25:15.209281 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:15.209260 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh" event={"ID":"e832e7ea-798d-488b-99e7-729ae596b1db","Type":"ContainerDied","Data":"c9f4415fbaf8664d4110e55d7a6d531110d180379f3d2dcd2adbbd5eda8ffb32"} Jul 22 09:25:15.209389 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:15.209290 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh" Jul 22 09:25:15.209389 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:15.209295 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9f4415fbaf8664d4110e55d7a6d531110d180379f3d2dcd2adbbd5eda8ffb32" Jul 22 09:25:22.307866 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:22.307832 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:26.013452 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:26.013419 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:30.253656 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:30.253611 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:31.701075 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:31.701036 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:31.971604 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:31.971513 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:34.513663 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:34.513625 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:35.114871 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:35.114813 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:35.546279 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:35.546246 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:35.890275 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:35.890173 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:36.030843 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:36.030815 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:39.115689 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:39.115657 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:39.389923 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:39.389828 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:39.597245 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:39.597194 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:39.794497 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:39.794465 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:39.958351 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:39.958299 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:40.223716 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:40.223613 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:42.571689 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:42.571661 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:42.758631 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:42.758597 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:44.060292 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:44.060261 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:44.244908 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:44.244867 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:46.526006 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:46.525970 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:46.654187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:46.654158 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:49.986003 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:49.985961 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:50.120820 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:50.120779 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:56.399696 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:56.399661 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:56.504504 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:56.504467 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:56.606370 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:56.606338 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:59.001059 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:59.001005 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:25:59.105470 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:25:59.105428 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:01.127280 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:01.127245 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:01.226551 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:01.226514 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:07.669954 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:07.669900 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:07.769960 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:07.769926 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:14.208007 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:14.207963 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:14.307979 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:14.307939 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:16.851553 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:16.851518 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:16.964722 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:16.964685 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:19.828780 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:19.828732 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:19.934936 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:19.934901 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:20.449947 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.449918 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv"] Jul 22 09:26:20.450201 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450188 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e832e7ea-798d-488b-99e7-729ae596b1db" containerName="extract" Jul 22 09:26:20.450255 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450204 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e832e7ea-798d-488b-99e7-729ae596b1db" containerName="extract" Jul 22 09:26:20.450255 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450216 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="481aaf1f-a26a-41d3-b266-e439c874e042" containerName="pull" Jul 22 09:26:20.450255 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450222 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="481aaf1f-a26a-41d3-b266-e439c874e042" containerName="pull" Jul 22 09:26:20.450255 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450229 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f2bc8a1-a99e-4238-9f60-24098f97e5d9" containerName="util" Jul 22 09:26:20.450255 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450235 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f2bc8a1-a99e-4238-9f60-24098f97e5d9" containerName="util" Jul 22 09:26:20.450255 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450241 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f2bc8a1-a99e-4238-9f60-24098f97e5d9" containerName="pull" Jul 22 09:26:20.450255 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450246 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f2bc8a1-a99e-4238-9f60-24098f97e5d9" containerName="pull" Jul 22 09:26:20.450255 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450252 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7434ca-9a9b-40d4-8219-d03a2e4a05f1" containerName="pull" Jul 22 09:26:20.450255 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450257 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7434ca-9a9b-40d4-8219-d03a2e4a05f1" containerName="pull" Jul 22 09:26:20.450567 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450264 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="481aaf1f-a26a-41d3-b266-e439c874e042" containerName="util" Jul 22 09:26:20.450567 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450269 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="481aaf1f-a26a-41d3-b266-e439c874e042" containerName="util" Jul 22 09:26:20.450567 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450277 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f2bc8a1-a99e-4238-9f60-24098f97e5d9" containerName="extract" Jul 22 09:26:20.450567 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450282 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f2bc8a1-a99e-4238-9f60-24098f97e5d9" containerName="extract" Jul 22 09:26:20.450567 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450290 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e832e7ea-798d-488b-99e7-729ae596b1db" containerName="pull" Jul 22 09:26:20.450567 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450295 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e832e7ea-798d-488b-99e7-729ae596b1db" containerName="pull" Jul 22 09:26:20.450567 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450320 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="481aaf1f-a26a-41d3-b266-e439c874e042" containerName="extract" Jul 22 09:26:20.450567 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450326 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="481aaf1f-a26a-41d3-b266-e439c874e042" containerName="extract" Jul 22 09:26:20.450567 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450333 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e832e7ea-798d-488b-99e7-729ae596b1db" containerName="util" Jul 22 09:26:20.450567 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450337 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e832e7ea-798d-488b-99e7-729ae596b1db" containerName="util" Jul 22 09:26:20.450567 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450342 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7434ca-9a9b-40d4-8219-d03a2e4a05f1" containerName="util" Jul 22 09:26:20.450567 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450347 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7434ca-9a9b-40d4-8219-d03a2e4a05f1" containerName="util" Jul 22 09:26:20.450567 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450353 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7434ca-9a9b-40d4-8219-d03a2e4a05f1" containerName="extract" Jul 22 09:26:20.450567 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450358 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7434ca-9a9b-40d4-8219-d03a2e4a05f1" containerName="extract" Jul 22 09:26:20.450567 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450408 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f7434ca-9a9b-40d4-8219-d03a2e4a05f1" containerName="extract" Jul 22 09:26:20.450567 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450414 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="e832e7ea-798d-488b-99e7-729ae596b1db" containerName="extract" Jul 22 09:26:20.450567 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450420 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f2bc8a1-a99e-4238-9f60-24098f97e5d9" containerName="extract" Jul 22 09:26:20.450567 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.450426 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="481aaf1f-a26a-41d3-b266-e439c874e042" containerName="extract" Jul 22 09:26:20.453233 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.453217 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.457949 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.457930 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-n6b22\"" Jul 22 09:26:20.469679 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.469657 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv"] Jul 22 09:26:20.477761 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.477710 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:20.566422 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.566382 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.566422 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.566431 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.566634 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.566524 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.566634 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.566555 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.566634 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.566577 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.566634 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.566615 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hn9g\" (UniqueName: \"kubernetes.io/projected/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-kube-api-access-4hn9g\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.566754 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.566645 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.566754 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.566664 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.566754 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.566682 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.667345 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.667296 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.667345 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.667344 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.667589 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.667465 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.667589 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.667502 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.667698 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.667629 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.667698 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.667670 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.667801 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.667753 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.667848 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.667794 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.667902 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.667854 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hn9g\" (UniqueName: \"kubernetes.io/projected/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-kube-api-access-4hn9g\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.667902 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.667894 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.668010 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.667946 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.668132 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.668112 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.668211 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.668202 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.668254 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.668201 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.669715 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.669691 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.669855 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.669836 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.676516 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.676491 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.676622 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.676606 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hn9g\" (UniqueName: \"kubernetes.io/projected/f19d7a4f-3cb5-4652-b022-8c660e2f7c2f-kube-api-access-4hn9g\") pod \"maas-default-gateway-openshift-default-56654dcf7-2tstv\" (UID: \"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.763528 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.763477 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:20.897830 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.897803 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv"] Jul 22 09:26:20.900255 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:26:20.900222 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19d7a4f_3cb5_4652_b022_8c660e2f7c2f.slice/crio-cc8b125c676fe220d4482d5dacd6b1faa4094113f82f510c19170535613f9541 WatchSource:0}: Error finding container cc8b125c676fe220d4482d5dacd6b1faa4094113f82f510c19170535613f9541: Status 404 returned error can't find the container with id cc8b125c676fe220d4482d5dacd6b1faa4094113f82f510c19170535613f9541 Jul 22 09:26:20.900840 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.900811 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:20.902749 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.902718 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:26:20.902825 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.902783 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:26:20.902825 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:20.902811 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:26:21.505056 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:21.505016 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" event={"ID":"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f","Type":"ContainerStarted","Data":"edc4883d32cb80e0900279713d9853f061021aa8cbe4038a3ee92f103d9ae32a"} Jul 22 09:26:21.505240 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:21.505111 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" event={"ID":"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f","Type":"ContainerStarted","Data":"cc8b125c676fe220d4482d5dacd6b1faa4094113f82f510c19170535613f9541"} Jul 22 09:26:21.528834 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:21.528786 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" podStartSLOduration=1.5287716759999999 podStartE2EDuration="1.528771676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:26:21.526960824 +0000 UTC m=+511.990755274" watchObservedRunningTime="2026-07-22 09:26:21.528771676 +0000 UTC m=+511.992566114" Jul 22 09:26:21.764501 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:21.764424 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:21.769122 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:21.769099 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:22.509524 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:22.509496 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:22.510337 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:22.510318 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:26:22.636659 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:22.636600 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:22.749272 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:22.749237 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:22.812605 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:22.812570 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:23.501952 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:23.501917 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:23.601226 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:23.601189 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:26.688262 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:26.688221 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:26.787364 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:26.787328 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:33.243023 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:33.242986 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:33.344011 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:33.343972 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:43.768196 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:43.768112 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:43.866846 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:43.866804 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:44.283515 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:44.283480 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:56.540377 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:56.540338 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:56.710943 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:56.710910 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:26:57.215436 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:26:57.215403 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:27:03.183421 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:03.183371 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:27:03.282129 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:03.282088 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:27:03.371864 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:03.371831 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:27:03.710120 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:03.710085 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:27:07.157443 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:07.157408 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:27:07.264760 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:07.264721 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:27:39.963045 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:39.963004 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:27:40.049113 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:40.049078 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:27:40.359668 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:40.359637 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:27:40.457566 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:40.457532 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-857c48d78c-4gkbw"] Jul 22 09:27:40.460644 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:40.460627 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-857c48d78c-4gkbw" Jul 22 09:27:40.466275 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:40.466253 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 22 09:27:40.466573 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:40.466543 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:27:40.467263 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:40.467010 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 22 09:27:40.467263 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:40.467029 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 09:27:40.475262 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:40.475241 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-857c48d78c-4gkbw"] Jul 22 09:27:40.509132 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:40.509104 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qn2f\" (UniqueName: \"kubernetes.io/projected/602dc950-1b12-4399-8459-939eae98762c-kube-api-access-5qn2f\") pod \"maas-api-857c48d78c-4gkbw\" (UID: \"602dc950-1b12-4399-8459-939eae98762c\") " pod="odh-ai-gateway-infra/maas-api-857c48d78c-4gkbw" Jul 22 09:27:40.509264 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:40.509151 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/602dc950-1b12-4399-8459-939eae98762c-openshift-service-ca\") pod \"maas-api-857c48d78c-4gkbw\" (UID: \"602dc950-1b12-4399-8459-939eae98762c\") " pod="odh-ai-gateway-infra/maas-api-857c48d78c-4gkbw" Jul 22 09:27:40.509264 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:40.509227 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/602dc950-1b12-4399-8459-939eae98762c-maas-api-tls\") pod \"maas-api-857c48d78c-4gkbw\" (UID: \"602dc950-1b12-4399-8459-939eae98762c\") " pod="odh-ai-gateway-infra/maas-api-857c48d78c-4gkbw" Jul 22 09:27:40.598467 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:40.598430 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:27:40.609776 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:40.609696 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qn2f\" (UniqueName: \"kubernetes.io/projected/602dc950-1b12-4399-8459-939eae98762c-kube-api-access-5qn2f\") pod \"maas-api-857c48d78c-4gkbw\" (UID: \"602dc950-1b12-4399-8459-939eae98762c\") " pod="odh-ai-gateway-infra/maas-api-857c48d78c-4gkbw" Jul 22 09:27:40.609776 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:40.609747 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/602dc950-1b12-4399-8459-939eae98762c-openshift-service-ca\") pod \"maas-api-857c48d78c-4gkbw\" (UID: \"602dc950-1b12-4399-8459-939eae98762c\") " pod="odh-ai-gateway-infra/maas-api-857c48d78c-4gkbw" Jul 22 09:27:40.609996 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:40.609786 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/602dc950-1b12-4399-8459-939eae98762c-maas-api-tls\") pod \"maas-api-857c48d78c-4gkbw\" (UID: \"602dc950-1b12-4399-8459-939eae98762c\") " pod="odh-ai-gateway-infra/maas-api-857c48d78c-4gkbw" Jul 22 09:27:40.610474 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:40.610452 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/602dc950-1b12-4399-8459-939eae98762c-openshift-service-ca\") pod \"maas-api-857c48d78c-4gkbw\" (UID: \"602dc950-1b12-4399-8459-939eae98762c\") " pod="odh-ai-gateway-infra/maas-api-857c48d78c-4gkbw" Jul 22 09:27:40.612123 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:40.612102 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/602dc950-1b12-4399-8459-939eae98762c-maas-api-tls\") pod \"maas-api-857c48d78c-4gkbw\" (UID: \"602dc950-1b12-4399-8459-939eae98762c\") " pod="odh-ai-gateway-infra/maas-api-857c48d78c-4gkbw" Jul 22 09:27:40.621481 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:40.621462 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qn2f\" (UniqueName: \"kubernetes.io/projected/602dc950-1b12-4399-8459-939eae98762c-kube-api-access-5qn2f\") pod \"maas-api-857c48d78c-4gkbw\" (UID: \"602dc950-1b12-4399-8459-939eae98762c\") " pod="odh-ai-gateway-infra/maas-api-857c48d78c-4gkbw" Jul 22 09:27:40.770542 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:40.770498 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-857c48d78c-4gkbw" Jul 22 09:27:40.910604 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:40.910577 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-857c48d78c-4gkbw"] Jul 22 09:27:40.912998 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:27:40.912974 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602dc950_1b12_4399_8459_939eae98762c.slice/crio-ac2d5a69406c0c97577b7bdc917c6a102513a04f54e3a5db1be9001c7e352abf WatchSource:0}: Error finding container ac2d5a69406c0c97577b7bdc917c6a102513a04f54e3a5db1be9001c7e352abf: Status 404 returned error can't find the container with id ac2d5a69406c0c97577b7bdc917c6a102513a04f54e3a5db1be9001c7e352abf Jul 22 09:27:41.048959 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:41.048904 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:27:41.157187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:41.157154 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:27:41.352797 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:41.352769 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-vjmpp"] Jul 22 09:27:41.357036 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:41.357019 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-vjmpp" Jul 22 09:27:41.359768 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:41.359737 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-j6jlh\"" Jul 22 09:27:41.360189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:41.360165 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 22 09:27:41.370707 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:41.370685 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-vjmpp"] Jul 22 09:27:41.415894 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:41.415868 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zxmt\" (UniqueName: \"kubernetes.io/projected/fccc33db-070e-4ab7-a476-8882378a770b-kube-api-access-2zxmt\") pod \"payload-processing-5bf47cd988-vjmpp\" (UID: \"fccc33db-070e-4ab7-a476-8882378a770b\") " pod="openshift-ingress/payload-processing-5bf47cd988-vjmpp" Jul 22 09:27:41.416056 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:41.415936 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccc33db-070e-4ab7-a476-8882378a770b-plugins-config-volume\") pod \"payload-processing-5bf47cd988-vjmpp\" (UID: \"fccc33db-070e-4ab7-a476-8882378a770b\") " pod="openshift-ingress/payload-processing-5bf47cd988-vjmpp" Jul 22 09:27:41.516998 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:41.516956 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccc33db-070e-4ab7-a476-8882378a770b-plugins-config-volume\") pod \"payload-processing-5bf47cd988-vjmpp\" (UID: \"fccc33db-070e-4ab7-a476-8882378a770b\") " pod="openshift-ingress/payload-processing-5bf47cd988-vjmpp" Jul 22 09:27:41.517177 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:41.517044 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zxmt\" (UniqueName: \"kubernetes.io/projected/fccc33db-070e-4ab7-a476-8882378a770b-kube-api-access-2zxmt\") pod \"payload-processing-5bf47cd988-vjmpp\" (UID: \"fccc33db-070e-4ab7-a476-8882378a770b\") " pod="openshift-ingress/payload-processing-5bf47cd988-vjmpp" Jul 22 09:27:41.517798 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:41.517769 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccc33db-070e-4ab7-a476-8882378a770b-plugins-config-volume\") pod \"payload-processing-5bf47cd988-vjmpp\" (UID: \"fccc33db-070e-4ab7-a476-8882378a770b\") " pod="openshift-ingress/payload-processing-5bf47cd988-vjmpp" Jul 22 09:27:41.527204 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:41.527180 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zxmt\" (UniqueName: \"kubernetes.io/projected/fccc33db-070e-4ab7-a476-8882378a770b-kube-api-access-2zxmt\") pod \"payload-processing-5bf47cd988-vjmpp\" (UID: \"fccc33db-070e-4ab7-a476-8882378a770b\") " pod="openshift-ingress/payload-processing-5bf47cd988-vjmpp" Jul 22 09:27:41.668341 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:41.668015 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-vjmpp" Jul 22 09:27:41.828506 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:41.828036 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-vjmpp"] Jul 22 09:27:41.832151 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:27:41.832113 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfccc33db_070e_4ab7_a476_8882378a770b.slice/crio-a6f0065f006ee5111de244f3d49dd53babd1bf012645878bcfb61176ad003394 WatchSource:0}: Error finding container a6f0065f006ee5111de244f3d49dd53babd1bf012645878bcfb61176ad003394: Status 404 returned error can't find the container with id a6f0065f006ee5111de244f3d49dd53babd1bf012645878bcfb61176ad003394 Jul 22 09:27:41.866397 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:41.866349 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-vjmpp" event={"ID":"fccc33db-070e-4ab7-a476-8882378a770b","Type":"ContainerStarted","Data":"a6f0065f006ee5111de244f3d49dd53babd1bf012645878bcfb61176ad003394"} Jul 22 09:27:41.868243 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:41.868207 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-857c48d78c-4gkbw" event={"ID":"602dc950-1b12-4399-8459-939eae98762c","Type":"ContainerStarted","Data":"ac2d5a69406c0c97577b7bdc917c6a102513a04f54e3a5db1be9001c7e352abf"} Jul 22 09:27:44.906826 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:44.906783 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-857c48d78c-4gkbw" event={"ID":"602dc950-1b12-4399-8459-939eae98762c","Type":"ContainerStarted","Data":"393953241cec8d75cedc801b0dfa28e6fd5913635092ccbe3c0a251a5724ebae"} Jul 22 09:27:44.907291 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:44.907047 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-857c48d78c-4gkbw" Jul 22 09:27:44.908806 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:44.908786 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-vjmpp" event={"ID":"fccc33db-070e-4ab7-a476-8882378a770b","Type":"ContainerStarted","Data":"2b3a14e0ec2f3dc149fccddc03f217a3e69d72f1180424d8ff0f16908beb6219"} Jul 22 09:27:44.908928 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:44.908905 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-vjmpp" Jul 22 09:27:44.925363 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:44.925299 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-857c48d78c-4gkbw" podStartSLOduration=1.484816204 podStartE2EDuration="4.925286594s" totalImagesPullingTime="3.44047039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:27:44.924047123 +0000 UTC m=+595.387841593" watchObservedRunningTime="2026-07-22 09:27:44.925286594 +0000 UTC m=+595.389081079" Jul 22 09:27:44.942585 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:44.942544 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-vjmpp" podStartSLOduration=1.422330743 podStartE2EDuration="3.942531652s" totalImagesPullingTime="2.520200909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:27:44.940676398 +0000 UTC m=+595.404470836" watchObservedRunningTime="2026-07-22 09:27:44.942531652 +0000 UTC m=+595.406326090" Jul 22 09:27:48.929835 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:48.929752 2549 generic.go:356] "Generic (PLEG): container finished" podID="f19d7a4f-3cb5-4652-b022-8c660e2f7c2f" containerID="edc4883d32cb80e0900279713d9853f061021aa8cbe4038a3ee92f103d9ae32a" exitCode=0 Jul 22 09:27:48.930233 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:48.929826 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" event={"ID":"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f","Type":"ContainerDied","Data":"edc4883d32cb80e0900279713d9853f061021aa8cbe4038a3ee92f103d9ae32a"} Jul 22 09:27:48.930233 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:48.930189 2549 scope.go:117] "RemoveContainer" containerID="edc4883d32cb80e0900279713d9853f061021aa8cbe4038a3ee92f103d9ae32a" Jul 22 09:27:48.931174 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:48.931140 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:27:48.931268 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:48.931249 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:27:48.931339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:48.931324 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:27:49.936819 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:49.936782 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" event={"ID":"f19d7a4f-3cb5-4652-b022-8c660e2f7c2f","Type":"ContainerStarted","Data":"89fb2b212f8bebbddf09dbba9f52ba4c335921489eeada4eaed62e4ce27129b9"} Jul 22 09:27:50.259535 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:50.259459 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcsgw_dc8d8789-74a0-44ec-8902-44a9650c41ac/ovn-acl-logging/0.log" Jul 22 09:27:50.259701 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:50.259630 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcsgw_dc8d8789-74a0-44ec-8902-44a9650c41ac/ovn-acl-logging/0.log" Jul 22 09:27:50.765390 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:50.765351 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:27:50.765594 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:50.765485 2549 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-2tstv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 22 09:27:50.765594 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:50.765545 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" podUID="f19d7a4f-3cb5-4652-b022-8c660e2f7c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 22 09:27:50.918948 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:50.918921 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-857c48d78c-4gkbw" Jul 22 09:27:51.763936 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:51.763893 2549 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-2tstv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 22 09:27:51.764441 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:51.763960 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" podUID="f19d7a4f-3cb5-4652-b022-8c660e2f7c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 22 09:27:51.931540 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:51.931510 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:27:52.509699 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:52.509661 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:27:52.764825 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:52.764716 2549 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-2tstv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 22 09:27:52.764825 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:52.764805 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" podUID="f19d7a4f-3cb5-4652-b022-8c660e2f7c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 22 09:27:53.183837 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:53.183795 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:27:53.764627 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:53.764587 2549 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-2tstv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 22 09:27:53.764823 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:53.764650 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" podUID="f19d7a4f-3cb5-4652-b022-8c660e2f7c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 22 09:27:54.768255 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:54.768226 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:27:54.769234 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:54.769213 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2tstv" Jul 22 09:27:55.916234 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:55.916205 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-vjmpp" Jul 22 09:27:59.804490 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:27:59.804458 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:28:11.165893 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.165854 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5"] Jul 22 09:28:11.172447 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.172424 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:11.176370 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.176343 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 09:28:11.176370 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.176367 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 22 09:28:11.176522 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.176344 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 09:28:11.176522 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.176343 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xgb7q\"" Jul 22 09:28:11.183016 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.182993 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5"] Jul 22 09:28:11.251940 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.251905 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:28:11.275660 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.275633 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cdd9b07-f921-4eca-b431-69039ac89c89-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2gnf5\" (UID: \"1cdd9b07-f921-4eca-b431-69039ac89c89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:11.275808 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.275665 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cdd9b07-f921-4eca-b431-69039ac89c89-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2gnf5\" (UID: \"1cdd9b07-f921-4eca-b431-69039ac89c89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:11.275808 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.275692 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdd9b07-f921-4eca-b431-69039ac89c89-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2gnf5\" (UID: \"1cdd9b07-f921-4eca-b431-69039ac89c89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:11.275808 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.275708 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2svqm\" (UniqueName: \"kubernetes.io/projected/1cdd9b07-f921-4eca-b431-69039ac89c89-kube-api-access-2svqm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2gnf5\" (UID: \"1cdd9b07-f921-4eca-b431-69039ac89c89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:11.275808 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.275776 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cdd9b07-f921-4eca-b431-69039ac89c89-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2gnf5\" (UID: \"1cdd9b07-f921-4eca-b431-69039ac89c89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:11.275969 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.275807 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cdd9b07-f921-4eca-b431-69039ac89c89-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2gnf5\" (UID: \"1cdd9b07-f921-4eca-b431-69039ac89c89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:11.376098 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.376061 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cdd9b07-f921-4eca-b431-69039ac89c89-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2gnf5\" (UID: \"1cdd9b07-f921-4eca-b431-69039ac89c89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:11.376098 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.376101 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cdd9b07-f921-4eca-b431-69039ac89c89-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2gnf5\" (UID: \"1cdd9b07-f921-4eca-b431-69039ac89c89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:11.376372 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.376129 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cdd9b07-f921-4eca-b431-69039ac89c89-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2gnf5\" (UID: \"1cdd9b07-f921-4eca-b431-69039ac89c89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:11.376372 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.376269 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cdd9b07-f921-4eca-b431-69039ac89c89-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2gnf5\" (UID: \"1cdd9b07-f921-4eca-b431-69039ac89c89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:11.376372 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.376343 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdd9b07-f921-4eca-b431-69039ac89c89-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2gnf5\" (UID: \"1cdd9b07-f921-4eca-b431-69039ac89c89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:11.376530 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.376373 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2svqm\" (UniqueName: \"kubernetes.io/projected/1cdd9b07-f921-4eca-b431-69039ac89c89-kube-api-access-2svqm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2gnf5\" (UID: \"1cdd9b07-f921-4eca-b431-69039ac89c89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:11.376530 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.376518 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cdd9b07-f921-4eca-b431-69039ac89c89-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2gnf5\" (UID: \"1cdd9b07-f921-4eca-b431-69039ac89c89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:11.376639 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.376609 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cdd9b07-f921-4eca-b431-69039ac89c89-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2gnf5\" (UID: \"1cdd9b07-f921-4eca-b431-69039ac89c89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:11.376682 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.376644 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cdd9b07-f921-4eca-b431-69039ac89c89-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2gnf5\" (UID: \"1cdd9b07-f921-4eca-b431-69039ac89c89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:11.378433 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.378413 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cdd9b07-f921-4eca-b431-69039ac89c89-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2gnf5\" (UID: \"1cdd9b07-f921-4eca-b431-69039ac89c89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:11.378659 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.378642 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdd9b07-f921-4eca-b431-69039ac89c89-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2gnf5\" (UID: \"1cdd9b07-f921-4eca-b431-69039ac89c89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:11.389920 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.389896 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2svqm\" (UniqueName: \"kubernetes.io/projected/1cdd9b07-f921-4eca-b431-69039ac89c89-kube-api-access-2svqm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2gnf5\" (UID: \"1cdd9b07-f921-4eca-b431-69039ac89c89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:11.482751 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.482666 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:11.612914 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.612879 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5"] Jul 22 09:28:11.615986 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:28:11.615963 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cdd9b07_f921_4eca_b431_69039ac89c89.slice/crio-420a4a47ac6db779e1677bf45c8ba18c5dac585c718826259c9109cd482243ff WatchSource:0}: Error finding container 420a4a47ac6db779e1677bf45c8ba18c5dac585c718826259c9109cd482243ff: Status 404 returned error can't find the container with id 420a4a47ac6db779e1677bf45c8ba18c5dac585c718826259c9109cd482243ff Jul 22 09:28:11.617343 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:11.617327 2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 09:28:12.040520 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:12.040481 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" event={"ID":"1cdd9b07-f921-4eca-b431-69039ac89c89","Type":"ContainerStarted","Data":"420a4a47ac6db779e1677bf45c8ba18c5dac585c718826259c9109cd482243ff"} Jul 22 09:28:13.046812 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:13.046781 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" event={"ID":"1cdd9b07-f921-4eca-b431-69039ac89c89","Type":"ContainerStarted","Data":"d85e51e35c786c8cabc1537b74db6c2b3b9c29da3fefd4c1c3ef0f597590cd26"} Jul 22 09:28:13.067388 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:13.067342 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" podStartSLOduration=1.044407423 podStartE2EDuration="2.067327488s" totalImagesPullingTime="1.022920065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:28:13.066170296 +0000 UTC m=+623.529964735" watchObservedRunningTime="2026-07-22 09:28:13.067327488 +0000 UTC m=+623.531121920" Jul 22 09:28:18.409483 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:18.409449 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:28:19.517709 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:19.517680 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:28:21.483158 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:21.483125 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:21.483578 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:21.483421 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:21.495835 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:21.495811 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:22.100927 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:22.100900 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2gnf5" Jul 22 09:28:39.004129 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:39.004088 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:28:58.902024 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:28:58.901971 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:29:04.595770 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:29:04.595731 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:29:43.637135 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:29:43.637051 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:29:58.923412 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:29:58.923365 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:29:59.025068 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:29:59.025029 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:30:00.196609 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:30:00.196576 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:30:00.222070 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:30:00.222040 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745210-d7b6r"] Jul 22 09:30:00.225172 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:30:00.225152 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745210-d7b6r" Jul 22 09:30:00.228419 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:30:00.228400 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmn2j\"" Jul 22 09:30:00.234136 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:30:00.234119 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745210-d7b6r" Jul 22 09:30:00.240514 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:30:00.240486 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745210-d7b6r"] Jul 22 09:30:00.307775 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:30:00.307739 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:30:00.369068 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:30:00.369044 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745210-d7b6r"] Jul 22 09:30:00.371401 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:30:00.371372 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bc50eb_8892_403b_9efe_f89f89dd6341.slice/crio-96f2048bbe690880320ce60799e0956cc371016dad69d2f2661b9ce8ee99193e WatchSource:0}: Error finding container 96f2048bbe690880320ce60799e0956cc371016dad69d2f2661b9ce8ee99193e: Status 404 returned error can't find the container with id 96f2048bbe690880320ce60799e0956cc371016dad69d2f2661b9ce8ee99193e Jul 22 09:30:00.559206 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:30:00.559171 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745210-d7b6r" event={"ID":"70bc50eb-8892-403b-9efe-f89f89dd6341","Type":"ContainerStarted","Data":"96f2048bbe690880320ce60799e0956cc371016dad69d2f2661b9ce8ee99193e"} Jul 22 09:30:01.391726 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:30:01.391689 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:30:01.521448 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:30:01.521401 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:30:03.575658 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:30:03.575565 2549 generic.go:356] "Generic (PLEG): container finished" podID="70bc50eb-8892-403b-9efe-f89f89dd6341" containerID="38957a6d6c1f57cc9a43601ce43c156bcbe6cdff4656f27f3b6a05c5f82ffc81" exitCode=0 Jul 22 09:30:03.575658 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:30:03.575628 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745210-d7b6r" event={"ID":"70bc50eb-8892-403b-9efe-f89f89dd6341","Type":"ContainerDied","Data":"38957a6d6c1f57cc9a43601ce43c156bcbe6cdff4656f27f3b6a05c5f82ffc81"} Jul 22 09:30:04.710606 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:30:04.710581 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745210-d7b6r" Jul 22 09:30:05.585615 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:30:05.585588 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745210-d7b6r" Jul 22 09:30:05.585615 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:30:05.585606 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745210-d7b6r" event={"ID":"70bc50eb-8892-403b-9efe-f89f89dd6341","Type":"ContainerDied","Data":"96f2048bbe690880320ce60799e0956cc371016dad69d2f2661b9ce8ee99193e"} Jul 22 09:30:05.585807 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:30:05.585650 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96f2048bbe690880320ce60799e0956cc371016dad69d2f2661b9ce8ee99193e" Jul 22 09:31:09.553485 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:31:09.553431 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:31:09.613456 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:31:09.613421 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:32:05.429673 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:32:05.429632 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:32:06.666268 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:32:06.666218 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:32:16.878727 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:32:16.878692 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:32:42.220380 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:32:42.220265 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:32:42.318255 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:32:42.318224 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:32:42.426390 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:32:42.426349 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:32:48.215036 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:32:48.215000 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:32:48.274786 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:32:48.274756 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:32:48.365693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:32:48.365658 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:32:48.468944 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:32:48.468858 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:32:48.572853 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:32:48.572817 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:32:50.289187 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:32:50.289159 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcsgw_dc8d8789-74a0-44ec-8902-44a9650c41ac/ovn-acl-logging/0.log" Jul 22 09:32:50.304844 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:32:50.304823 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcsgw_dc8d8789-74a0-44ec-8902-44a9650c41ac/ovn-acl-logging/0.log" Jul 22 09:32:54.707883 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:32:54.707846 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:32:56.321542 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:32:56.321499 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:33:21.451917 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:33:21.451882 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:33:21.546775 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:33:21.546737 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:34:56.773248 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:34:56.773210 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:37:50.318954 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:37:50.318835 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcsgw_dc8d8789-74a0-44ec-8902-44a9650c41ac/ovn-acl-logging/0.log" Jul 22 09:37:50.334824 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:37:50.334797 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcsgw_dc8d8789-74a0-44ec-8902-44a9650c41ac/ovn-acl-logging/0.log" Jul 22 09:39:48.760753 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:39:48.760713 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:40:26.896081 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:40:26.895992 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:40:31.852128 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:40:31.852088 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:40:48.410056 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:40:48.410021 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:41:09.661896 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:41:09.661863 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:41:10.852548 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:41:10.852516 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:42:50.354745 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:42:50.354643 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcsgw_dc8d8789-74a0-44ec-8902-44a9650c41ac/ovn-acl-logging/0.log" Jul 22 09:42:50.371363 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:42:50.371341 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcsgw_dc8d8789-74a0-44ec-8902-44a9650c41ac/ovn-acl-logging/0.log" Jul 22 09:43:42.158335 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:43:42.158206 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:43:52.009596 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:43:52.009553 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:44:22.864574 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:22.864542 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:44:44.169921 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:44.169828 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:44:46.964332 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:46.964262 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:44:47.367711 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:47.367674 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:44:47.467450 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:47.467418 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:44:53.361812 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:53.361775 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:44:53.889438 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:53.889407 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t"] Jul 22 09:44:53.889715 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:53.889703 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70bc50eb-8892-403b-9efe-f89f89dd6341" containerName="cleanup" Jul 22 09:44:53.889768 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:53.889716 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="70bc50eb-8892-403b-9efe-f89f89dd6341" containerName="cleanup" Jul 22 09:44:53.889803 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:53.889786 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="70bc50eb-8892-403b-9efe-f89f89dd6341" containerName="cleanup" Jul 22 09:44:53.892724 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:53.892708 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:53.896230 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:53.896206 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1d701482-openshift-default-dockercfg-ztsbb\"" Jul 22 09:44:53.917934 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:53.917900 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t"] Jul 22 09:44:54.017961 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.017915 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-token\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.017961 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.017963 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-podinfo\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.018199 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.017986 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-envoy\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.018199 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.018046 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd32224f-90ac-44d5-b492-39d31c6e5cad-istiod-ca-cert\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.018199 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.018112 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-credential-socket\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.018199 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.018134 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-workload-socket\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.018199 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.018159 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-data\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.018199 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.018177 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-workload-certs\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.018199 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.018195 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lglxf\" (UniqueName: \"kubernetes.io/projected/bd32224f-90ac-44d5-b492-39d31c6e5cad-kube-api-access-lglxf\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.118780 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.118731 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-token\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.118780 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.118786 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-podinfo\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.119026 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.118819 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-envoy\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.119026 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.118840 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd32224f-90ac-44d5-b492-39d31c6e5cad-istiod-ca-cert\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.119026 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.118881 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-credential-socket\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.119202 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.119015 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-workload-socket\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.119202 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.119078 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-data\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.119202 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.119112 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-workload-certs\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.119202 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.119144 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lglxf\" (UniqueName: \"kubernetes.io/projected/bd32224f-90ac-44d5-b492-39d31c6e5cad-kube-api-access-lglxf\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.119495 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.119327 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-credential-socket\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.119495 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.119418 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-workload-socket\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.119611 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.119500 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-data\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.119611 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.119515 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-workload-certs\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.119720 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.119680 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd32224f-90ac-44d5-b492-39d31c6e5cad-istiod-ca-cert\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.121034 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.121015 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-envoy\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.121532 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.121515 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-podinfo\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.129668 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.129646 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-token\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.136409 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.136377 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lglxf\" (UniqueName: \"kubernetes.io/projected/bd32224f-90ac-44d5-b492-39d31c6e5cad-kube-api-access-lglxf\") pod \"e2e-ait-1d701482-openshift-default-6975647785-2wr5t\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.203324 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.203211 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:54.340887 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:44:54.340861 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd32224f_90ac_44d5_b492_39d31c6e5cad.slice/crio-eb748d3d9ef8932cb7c7a670dba6e6eaaf25e72b8e8f9fd26bfab3d2403950d8 WatchSource:0}: Error finding container eb748d3d9ef8932cb7c7a670dba6e6eaaf25e72b8e8f9fd26bfab3d2403950d8: Status 404 returned error can't find the container with id eb748d3d9ef8932cb7c7a670dba6e6eaaf25e72b8e8f9fd26bfab3d2403950d8 Jul 22 09:44:54.341005 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.340975 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t"] Jul 22 09:44:54.342807 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.342782 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:44:54.342877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.342840 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:44:54.342877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.342867 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:44:54.791365 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.791332 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" event={"ID":"bd32224f-90ac-44d5-b492-39d31c6e5cad","Type":"ContainerStarted","Data":"6df4e5501d50d75cf81e0aeb3585cc5418a86f538ded9d0b5ee40efe94740a33"} Jul 22 09:44:54.791778 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.791384 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" event={"ID":"bd32224f-90ac-44d5-b492-39d31c6e5cad","Type":"ContainerStarted","Data":"eb748d3d9ef8932cb7c7a670dba6e6eaaf25e72b8e8f9fd26bfab3d2403950d8"} Jul 22 09:44:54.836036 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:54.835985 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" podStartSLOduration=1.8359690629999998 podStartE2EDuration="1.835969063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:44:54.834529904 +0000 UTC m=+1625.298324342" watchObservedRunningTime="2026-07-22 09:44:54.835969063 +0000 UTC m=+1625.299763501" Jul 22 09:44:55.169661 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:55.169582 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:44:55.203722 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:55.203686 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:55.208549 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:55.208526 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:55.796538 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:55.796506 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:55.797490 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:55.797472 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:44:56.262696 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:56.262620 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:44:56.344629 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:56.344596 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:44:56.468464 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:56.468435 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww"] Jul 22 09:44:56.471698 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:56.471679 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww" Jul 22 09:44:56.474601 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:56.474581 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-876r5\"" Jul 22 09:44:56.475725 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:56.475710 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1d701482-serving-cert\"" Jul 22 09:44:56.493551 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:56.493507 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww"] Jul 22 09:44:56.504642 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:56.504616 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:44:56.641324 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:56.641269 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz655\" (UniqueName: \"kubernetes.io/projected/3ae6841e-fe81-4853-b751-3dbca8bd6faf-kube-api-access-qz655\") pod \"maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww\" (UID: \"3ae6841e-fe81-4853-b751-3dbca8bd6faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww" Jul 22 09:44:56.641498 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:56.641334 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ae6841e-fe81-4853-b751-3dbca8bd6faf-maas-api-tls\") pod \"maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww\" (UID: \"3ae6841e-fe81-4853-b751-3dbca8bd6faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww" Jul 22 09:44:56.641498 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:56.641446 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae6841e-fe81-4853-b751-3dbca8bd6faf-openshift-service-ca\") pod \"maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww\" (UID: \"3ae6841e-fe81-4853-b751-3dbca8bd6faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww" Jul 22 09:44:56.730926 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:56.730890 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:44:56.742040 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:56.742006 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz655\" (UniqueName: \"kubernetes.io/projected/3ae6841e-fe81-4853-b751-3dbca8bd6faf-kube-api-access-qz655\") pod \"maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww\" (UID: \"3ae6841e-fe81-4853-b751-3dbca8bd6faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww" Jul 22 09:44:56.742182 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:56.742047 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ae6841e-fe81-4853-b751-3dbca8bd6faf-maas-api-tls\") pod \"maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww\" (UID: \"3ae6841e-fe81-4853-b751-3dbca8bd6faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww" Jul 22 09:44:56.742182 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:56.742119 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae6841e-fe81-4853-b751-3dbca8bd6faf-openshift-service-ca\") pod \"maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww\" (UID: \"3ae6841e-fe81-4853-b751-3dbca8bd6faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww" Jul 22 09:44:56.742788 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:56.742765 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae6841e-fe81-4853-b751-3dbca8bd6faf-openshift-service-ca\") pod \"maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww\" (UID: \"3ae6841e-fe81-4853-b751-3dbca8bd6faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww" Jul 22 09:44:56.744551 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:56.744532 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ae6841e-fe81-4853-b751-3dbca8bd6faf-maas-api-tls\") pod \"maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww\" (UID: \"3ae6841e-fe81-4853-b751-3dbca8bd6faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww" Jul 22 09:44:56.751722 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:56.751703 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz655\" (UniqueName: \"kubernetes.io/projected/3ae6841e-fe81-4853-b751-3dbca8bd6faf-kube-api-access-qz655\") pod \"maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww\" (UID: \"3ae6841e-fe81-4853-b751-3dbca8bd6faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww" Jul 22 09:44:56.781924 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:56.781895 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww" Jul 22 09:44:56.929826 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:56.929800 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww"] Jul 22 09:44:56.931155 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:44:56.931127 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae6841e_fe81_4853_b751_3dbca8bd6faf.slice/crio-c8eed3cf41e694b39dcab795f5c0763107df6cf51f4753957cf0992b6fd12e02 WatchSource:0}: Error finding container c8eed3cf41e694b39dcab795f5c0763107df6cf51f4753957cf0992b6fd12e02: Status 404 returned error can't find the container with id c8eed3cf41e694b39dcab795f5c0763107df6cf51f4753957cf0992b6fd12e02 Jul 22 09:44:56.932564 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:56.932543 2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 09:44:57.159767 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:57.159732 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:44:57.260777 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:57.260695 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:44:57.808170 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:57.808134 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww" event={"ID":"3ae6841e-fe81-4853-b751-3dbca8bd6faf","Type":"ContainerStarted","Data":"b115747db2028a4e67ec8834339ddd181931cbe0c02ab73fafa19ef9fff376bb"} Jul 22 09:44:57.808366 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:57.808195 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww" event={"ID":"3ae6841e-fe81-4853-b751-3dbca8bd6faf","Type":"ContainerStarted","Data":"c8eed3cf41e694b39dcab795f5c0763107df6cf51f4753957cf0992b6fd12e02"} Jul 22 09:44:57.808424 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:57.808366 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww" Jul 22 09:44:57.830428 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:44:57.830380 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww" podStartSLOduration=1.061424413 podStartE2EDuration="1.830365511s" totalImagesPullingTime="768.941098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:44:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:44:57.829494464 +0000 UTC m=+1628.293288905" watchObservedRunningTime="2026-07-22 09:44:57.830365511 +0000 UTC m=+1628.294159949" Jul 22 09:45:00.130272 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:00.130232 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745225-txqdx"] Jul 22 09:45:00.133658 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:00.133641 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745225-txqdx" Jul 22 09:45:00.135961 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:00.135938 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmn2j\"" Jul 22 09:45:00.141399 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:00.141009 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745225-txqdx"] Jul 22 09:45:00.145982 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:00.145960 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745225-txqdx" Jul 22 09:45:00.282175 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:00.282151 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745225-txqdx"] Jul 22 09:45:00.284013 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:45:00.283984 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode811da67_1c91_42a2_9a49_725d8a247b3c.slice/crio-dad8f5eec0e8f5619ca7ddf1902d715a921ff5e3937a85cd69f5e614b24cf761 WatchSource:0}: Error finding container dad8f5eec0e8f5619ca7ddf1902d715a921ff5e3937a85cd69f5e614b24cf761: Status 404 returned error can't find the container with id dad8f5eec0e8f5619ca7ddf1902d715a921ff5e3937a85cd69f5e614b24cf761 Jul 22 09:45:00.825522 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:00.825493 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745225-txqdx" event={"ID":"e811da67-1c91-42a2-9a49-725d8a247b3c","Type":"ContainerStarted","Data":"2e9330198501cf5966b63c7e8abb7ffb1870fe0051307d0f99cc905a1f7b0ab0"} Jul 22 09:45:00.825691 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:00.825545 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745225-txqdx" event={"ID":"e811da67-1c91-42a2-9a49-725d8a247b3c","Type":"ContainerStarted","Data":"dad8f5eec0e8f5619ca7ddf1902d715a921ff5e3937a85cd69f5e614b24cf761"} Jul 22 09:45:00.842678 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:00.842635 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745225-txqdx" podStartSLOduration=0.842620947 podStartE2EDuration="842.620947ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:45:00.841045478 +0000 UTC m=+1631.304839930" watchObservedRunningTime="2026-07-22 09:45:00.842620947 +0000 UTC m=+1631.306415405" Jul 22 09:45:01.831983 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:01.831947 2549 generic.go:356] "Generic (PLEG): container finished" podID="e811da67-1c91-42a2-9a49-725d8a247b3c" containerID="2e9330198501cf5966b63c7e8abb7ffb1870fe0051307d0f99cc905a1f7b0ab0" exitCode=0 Jul 22 09:45:01.832427 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:01.832038 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745225-txqdx" event={"ID":"e811da67-1c91-42a2-9a49-725d8a247b3c","Type":"ContainerDied","Data":"2e9330198501cf5966b63c7e8abb7ffb1870fe0051307d0f99cc905a1f7b0ab0"} Jul 22 09:45:02.966984 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:02.966958 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745225-txqdx" Jul 22 09:45:03.818407 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:03.818377 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww" Jul 22 09:45:03.843352 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:03.843293 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745225-txqdx" event={"ID":"e811da67-1c91-42a2-9a49-725d8a247b3c","Type":"ContainerDied","Data":"dad8f5eec0e8f5619ca7ddf1902d715a921ff5e3937a85cd69f5e614b24cf761"} Jul 22 09:45:03.843503 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:03.843373 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dad8f5eec0e8f5619ca7ddf1902d715a921ff5e3937a85cd69f5e614b24cf761" Jul 22 09:45:03.843503 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:03.843332 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745225-txqdx" Jul 22 09:45:11.821570 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:11.820737 2549 patch_prober.go:28] interesting pod/e2e-ait-1d701482-openshift-default-6975647785-2wr5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 22 09:45:11.821570 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:11.820824 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" podUID="bd32224f-90ac-44d5-b492-39d31c6e5cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 22 09:45:11.889690 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:11.889658 2549 generic.go:356] "Generic (PLEG): container finished" podID="bd32224f-90ac-44d5-b492-39d31c6e5cad" containerID="6df4e5501d50d75cf81e0aeb3585cc5418a86f538ded9d0b5ee40efe94740a33" exitCode=0 Jul 22 09:45:11.889690 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:11.889696 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" event={"ID":"bd32224f-90ac-44d5-b492-39d31c6e5cad","Type":"ContainerDied","Data":"6df4e5501d50d75cf81e0aeb3585cc5418a86f538ded9d0b5ee40efe94740a33"} Jul 22 09:45:11.890069 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:11.890057 2549 scope.go:117] "RemoveContainer" containerID="6df4e5501d50d75cf81e0aeb3585cc5418a86f538ded9d0b5ee40efe94740a33" Jul 22 09:45:11.891019 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:11.890992 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:45:11.891090 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:11.891066 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:45:11.891127 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:11.891117 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:45:12.897104 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:12.897067 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" event={"ID":"bd32224f-90ac-44d5-b492-39d31c6e5cad","Type":"ContainerStarted","Data":"165cc489ce26d251f6cf9680b89ba73e08bcf5ca89f08003e68499c487605cd2"} Jul 22 09:45:13.203614 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:13.203521 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:45:13.203810 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:13.203787 2549 patch_prober.go:28] interesting pod/e2e-ait-1d701482-openshift-default-6975647785-2wr5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 22 09:45:13.203853 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:13.203838 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" podUID="bd32224f-90ac-44d5-b492-39d31c6e5cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 22 09:45:14.203642 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:14.203598 2549 patch_prober.go:28] interesting pod/e2e-ait-1d701482-openshift-default-6975647785-2wr5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 22 09:45:14.204118 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:14.203665 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" podUID="bd32224f-90ac-44d5-b492-39d31c6e5cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 22 09:45:15.204126 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:15.204090 2549 patch_prober.go:28] interesting pod/e2e-ait-1d701482-openshift-default-6975647785-2wr5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 22 09:45:15.204555 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:15.204148 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" podUID="bd32224f-90ac-44d5-b492-39d31c6e5cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 22 09:45:16.203852 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:16.203816 2549 patch_prober.go:28] interesting pod/e2e-ait-1d701482-openshift-default-6975647785-2wr5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 22 09:45:16.204099 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:16.203866 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" podUID="bd32224f-90ac-44d5-b492-39d31c6e5cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 22 09:45:17.207495 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:17.207464 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:45:17.207875 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:17.207685 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:45:17.208402 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:17.208376 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:45:21.674153 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:21.674118 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d701482-tlrmt"] Jul 22 09:45:21.674613 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:21.674586 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e811da67-1c91-42a2-9a49-725d8a247b3c" containerName="cleanup" Jul 22 09:45:21.674613 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:21.674605 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e811da67-1c91-42a2-9a49-725d8a247b3c" containerName="cleanup" Jul 22 09:45:21.674738 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:21.674704 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="e811da67-1c91-42a2-9a49-725d8a247b3c" containerName="cleanup" Jul 22 09:45:21.677607 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:21.677587 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d701482-tlrmt" Jul 22 09:45:21.679949 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:21.679928 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmn2j\"" Jul 22 09:45:21.685521 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:21.685441 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d701482-tlrmt"] Jul 22 09:45:21.765247 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:21.765219 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/542808f7-5af1-4845-b89d-1548d93fa647-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d701482-tlrmt\" (UID: \"542808f7-5af1-4845-b89d-1548d93fa647\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d701482-tlrmt" Jul 22 09:45:21.866388 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:21.866363 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/542808f7-5af1-4845-b89d-1548d93fa647-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d701482-tlrmt\" (UID: \"542808f7-5af1-4845-b89d-1548d93fa647\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d701482-tlrmt" Jul 22 09:45:21.866914 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:21.866896 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/542808f7-5af1-4845-b89d-1548d93fa647-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d701482-tlrmt\" (UID: \"542808f7-5af1-4845-b89d-1548d93fa647\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d701482-tlrmt" Jul 22 09:45:21.988716 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:21.988637 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d701482-tlrmt" Jul 22 09:45:22.116969 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:22.116944 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d701482-tlrmt"] Jul 22 09:45:22.118609 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:45:22.118582 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542808f7_5af1_4845_b89d_1548d93fa647.slice/crio-493cc7f1e80baee9d139f3473f84c2091d06579e061c0812acf46c837d5b7f2c WatchSource:0}: Error finding container 493cc7f1e80baee9d139f3473f84c2091d06579e061c0812acf46c837d5b7f2c: Status 404 returned error can't find the container with id 493cc7f1e80baee9d139f3473f84c2091d06579e061c0812acf46c837d5b7f2c Jul 22 09:45:22.946188 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:22.946157 2549 generic.go:356] "Generic (PLEG): container finished" podID="542808f7-5af1-4845-b89d-1548d93fa647" containerID="b5fa74fbf21b4945cefb0bb3ab72aba4e148345c67da7f0ded7e7aa361da93fa" exitCode=0 Jul 22 09:45:22.946649 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:22.946220 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d701482-tlrmt" event={"ID":"542808f7-5af1-4845-b89d-1548d93fa647","Type":"ContainerDied","Data":"b5fa74fbf21b4945cefb0bb3ab72aba4e148345c67da7f0ded7e7aa361da93fa"} Jul 22 09:45:22.946649 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:22.946261 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d701482-tlrmt" event={"ID":"542808f7-5af1-4845-b89d-1548d93fa647","Type":"ContainerStarted","Data":"493cc7f1e80baee9d139f3473f84c2091d06579e061c0812acf46c837d5b7f2c"} Jul 22 09:45:24.083357 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:24.083329 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d701482-tlrmt" Jul 22 09:45:24.185439 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:24.185408 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/542808f7-5af1-4845-b89d-1548d93fa647-maas-api-service-ca\") pod \"542808f7-5af1-4845-b89d-1548d93fa647\" (UID: \"542808f7-5af1-4845-b89d-1548d93fa647\") " Jul 22 09:45:24.185776 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:24.185751 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/542808f7-5af1-4845-b89d-1548d93fa647-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "542808f7-5af1-4845-b89d-1548d93fa647" (UID: "542808f7-5af1-4845-b89d-1548d93fa647"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:45:24.286589 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:24.286550 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/542808f7-5af1-4845-b89d-1548d93fa647-maas-api-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:45:24.956690 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:24.956662 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d701482-tlrmt" Jul 22 09:45:24.956841 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:24.956703 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d701482-tlrmt" event={"ID":"542808f7-5af1-4845-b89d-1548d93fa647","Type":"ContainerDied","Data":"493cc7f1e80baee9d139f3473f84c2091d06579e061c0812acf46c837d5b7f2c"} Jul 22 09:45:24.956841 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:24.956747 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="493cc7f1e80baee9d139f3473f84c2091d06579e061c0812acf46c837d5b7f2c" Jul 22 09:45:32.126227 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:32.126196 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww"] Jul 22 09:45:32.126618 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:32.126551 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww" podUID="3ae6841e-fe81-4853-b751-3dbca8bd6faf" containerName="maas-api" containerID="cri-o://b115747db2028a4e67ec8834339ddd181931cbe0c02ab73fafa19ef9fff376bb" gracePeriod=30 Jul 22 09:45:32.153038 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:32.153011 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:45:32.373591 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:32.373567 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww" Jul 22 09:45:32.456295 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:32.456209 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ae6841e-fe81-4853-b751-3dbca8bd6faf-maas-api-tls\") pod \"3ae6841e-fe81-4853-b751-3dbca8bd6faf\" (UID: \"3ae6841e-fe81-4853-b751-3dbca8bd6faf\") " Jul 22 09:45:32.456295 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:32.456260 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae6841e-fe81-4853-b751-3dbca8bd6faf-openshift-service-ca\") pod \"3ae6841e-fe81-4853-b751-3dbca8bd6faf\" (UID: \"3ae6841e-fe81-4853-b751-3dbca8bd6faf\") " Jul 22 09:45:32.456508 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:32.456343 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qz655\" (UniqueName: \"kubernetes.io/projected/3ae6841e-fe81-4853-b751-3dbca8bd6faf-kube-api-access-qz655\") pod \"3ae6841e-fe81-4853-b751-3dbca8bd6faf\" (UID: \"3ae6841e-fe81-4853-b751-3dbca8bd6faf\") " Jul 22 09:45:32.456687 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:32.456661 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae6841e-fe81-4853-b751-3dbca8bd6faf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3ae6841e-fe81-4853-b751-3dbca8bd6faf" (UID: "3ae6841e-fe81-4853-b751-3dbca8bd6faf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:45:32.458481 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:32.458455 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ae6841e-fe81-4853-b751-3dbca8bd6faf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3ae6841e-fe81-4853-b751-3dbca8bd6faf" (UID: "3ae6841e-fe81-4853-b751-3dbca8bd6faf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:45:32.458558 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:32.458503 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae6841e-fe81-4853-b751-3dbca8bd6faf-kube-api-access-qz655" (OuterVolumeSpecName: "kube-api-access-qz655") pod "3ae6841e-fe81-4853-b751-3dbca8bd6faf" (UID: "3ae6841e-fe81-4853-b751-3dbca8bd6faf"). InnerVolumeSpecName "kube-api-access-qz655". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:45:32.557022 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:32.556982 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ae6841e-fe81-4853-b751-3dbca8bd6faf-maas-api-tls\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:45:32.557192 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:32.557045 2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae6841e-fe81-4853-b751-3dbca8bd6faf-openshift-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:45:32.557192 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:32.557061 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz655\" (UniqueName: \"kubernetes.io/projected/3ae6841e-fe81-4853-b751-3dbca8bd6faf-kube-api-access-qz655\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:45:33.000684 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:33.000649 2549 generic.go:356] "Generic (PLEG): container finished" podID="3ae6841e-fe81-4853-b751-3dbca8bd6faf" containerID="b115747db2028a4e67ec8834339ddd181931cbe0c02ab73fafa19ef9fff376bb" exitCode=0 Jul 22 09:45:33.000860 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:33.000695 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww" Jul 22 09:45:33.000860 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:33.000734 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww" event={"ID":"3ae6841e-fe81-4853-b751-3dbca8bd6faf","Type":"ContainerDied","Data":"b115747db2028a4e67ec8834339ddd181931cbe0c02ab73fafa19ef9fff376bb"} Jul 22 09:45:33.000860 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:33.000791 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww" event={"ID":"3ae6841e-fe81-4853-b751-3dbca8bd6faf","Type":"ContainerDied","Data":"c8eed3cf41e694b39dcab795f5c0763107df6cf51f4753957cf0992b6fd12e02"} Jul 22 09:45:33.000860 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:33.000814 2549 scope.go:117] "RemoveContainer" containerID="b115747db2028a4e67ec8834339ddd181931cbe0c02ab73fafa19ef9fff376bb" Jul 22 09:45:33.010042 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:33.010025 2549 scope.go:117] "RemoveContainer" containerID="b115747db2028a4e67ec8834339ddd181931cbe0c02ab73fafa19ef9fff376bb" Jul 22 09:45:33.010295 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:45:33.010277 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b115747db2028a4e67ec8834339ddd181931cbe0c02ab73fafa19ef9fff376bb\": container with ID starting with b115747db2028a4e67ec8834339ddd181931cbe0c02ab73fafa19ef9fff376bb not found: ID does not exist" containerID="b115747db2028a4e67ec8834339ddd181931cbe0c02ab73fafa19ef9fff376bb" Jul 22 09:45:33.010364 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:33.010320 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b115747db2028a4e67ec8834339ddd181931cbe0c02ab73fafa19ef9fff376bb"} err="failed to get container status \"b115747db2028a4e67ec8834339ddd181931cbe0c02ab73fafa19ef9fff376bb\": rpc error: code = NotFound desc = could not find container \"b115747db2028a4e67ec8834339ddd181931cbe0c02ab73fafa19ef9fff376bb\": container with ID starting with b115747db2028a4e67ec8834339ddd181931cbe0c02ab73fafa19ef9fff376bb not found: ID does not exist" Jul 22 09:45:33.022060 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:33.022037 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww"] Jul 22 09:45:33.024039 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:33.024016 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d701482-5c8b8c7b77-gm5ww"] Jul 22 09:45:34.340240 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:34.340205 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ae6841e-fe81-4853-b751-3dbca8bd6faf" path="/var/lib/kubelet/pods/3ae6841e-fe81-4853-b751-3dbca8bd6faf/volumes" Jul 22 09:45:36.754078 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:36.754046 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:45:36.754925 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:36.754904 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d701482-tlrmt"] Jul 22 09:45:36.756941 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:36.756922 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d701482-tlrmt"] Jul 22 09:45:36.887169 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:36.887133 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t"] Jul 22 09:45:36.887495 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:36.887437 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" podUID="bd32224f-90ac-44d5-b492-39d31c6e5cad" containerName="istio-proxy" containerID="cri-o://165cc489ce26d251f6cf9680b89ba73e08bcf5ca89f08003e68499c487605cd2" gracePeriod=30 Jul 22 09:45:36.916476 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:36.916452 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:45:37.221641 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:37.221609 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:45:37.429987 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:37.429957 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:45:37.796571 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:37.796540 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:45:37.889188 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:37.889157 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:45:37.998069 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:37.998042 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r"] Jul 22 09:45:37.998400 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:37.998386 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="542808f7-5af1-4845-b89d-1548d93fa647" containerName="revoke-keys" Jul 22 09:45:37.998453 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:37.998401 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="542808f7-5af1-4845-b89d-1548d93fa647" containerName="revoke-keys" Jul 22 09:45:37.998453 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:37.998418 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ae6841e-fe81-4853-b751-3dbca8bd6faf" containerName="maas-api" Jul 22 09:45:37.998453 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:37.998424 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ae6841e-fe81-4853-b751-3dbca8bd6faf" containerName="maas-api" Jul 22 09:45:37.998562 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:37.998487 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="542808f7-5af1-4845-b89d-1548d93fa647" containerName="revoke-keys" Jul 22 09:45:37.998562 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:37.998498 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ae6841e-fe81-4853-b751-3dbca8bd6faf" containerName="maas-api" Jul 22 09:45:38.002613 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.002595 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.005007 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.004987 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-9e388426-gw-openshift-default-dockercfg-58tmg\"" Jul 22 09:45:38.018342 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.018279 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r"] Jul 22 09:45:38.032833 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.032807 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:45:38.110769 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.110682 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-workload-socket\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.110769 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.110724 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f92rc\" (UniqueName: \"kubernetes.io/projected/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-kube-api-access-f92rc\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.110970 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.110800 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-token\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.110970 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.110884 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-credential-socket\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.110970 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.110911 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-data\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.110970 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.110955 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-envoy\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.111129 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.111006 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-podinfo\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.111129 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.111066 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-workload-certs\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.111129 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.111099 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istiod-ca-cert\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.130197 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.130169 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:45:38.212435 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.212398 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-credential-socket\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.212435 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.212436 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-data\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.212690 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.212478 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-envoy\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.212690 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.212508 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-podinfo\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.212690 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.212570 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-workload-certs\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.212690 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.212624 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istiod-ca-cert\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.212690 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.212663 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-workload-socket\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.212690 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.212684 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f92rc\" (UniqueName: \"kubernetes.io/projected/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-kube-api-access-f92rc\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.213002 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.212729 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-token\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.213002 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.212863 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-credential-socket\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.213002 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.212934 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-workload-certs\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.213171 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.213028 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-workload-socket\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.213171 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.213142 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-data\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.213377 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.213359 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istiod-ca-cert\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.214870 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.214847 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-envoy\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.215132 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.215111 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-podinfo\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.221447 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.221428 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f92rc\" (UniqueName: \"kubernetes.io/projected/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-kube-api-access-f92rc\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.221550 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.221535 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-token\") pod \"e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.313192 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.313155 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:38.340721 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.340692 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="542808f7-5af1-4845-b89d-1548d93fa647" path="/var/lib/kubelet/pods/542808f7-5af1-4845-b89d-1548d93fa647/volumes" Jul 22 09:45:38.451477 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.451447 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r"] Jul 22 09:45:38.453035 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:45:38.453007 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de07f5e_bf70_4b82_825f_c7fa6bf6f945.slice/crio-0b2595c8344954c960a4dab1a84ac86772de351182449dcef732289cf1f76a20 WatchSource:0}: Error finding container 0b2595c8344954c960a4dab1a84ac86772de351182449dcef732289cf1f76a20: Status 404 returned error can't find the container with id 0b2595c8344954c960a4dab1a84ac86772de351182449dcef732289cf1f76a20 Jul 22 09:45:38.455479 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.455445 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:45:38.455549 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.455512 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:45:38.455549 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:38.455543 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:45:39.037214 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:39.037178 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" event={"ID":"5de07f5e-bf70-4b82-825f-c7fa6bf6f945","Type":"ContainerStarted","Data":"c4df1ca3f3094c4d7a78bb57cdd1c064ad78b683df122ba6ea3fa86735c85142"} Jul 22 09:45:39.037605 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:39.037233 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" event={"ID":"5de07f5e-bf70-4b82-825f-c7fa6bf6f945","Type":"ContainerStarted","Data":"0b2595c8344954c960a4dab1a84ac86772de351182449dcef732289cf1f76a20"} Jul 22 09:45:39.063714 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:39.063669 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" podStartSLOduration=2.0636565669999998 podStartE2EDuration="2.063656567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:45:39.061269041 +0000 UTC m=+1669.525063484" watchObservedRunningTime="2026-07-22 09:45:39.063656567 +0000 UTC m=+1669.527451004" Jul 22 09:45:39.313545 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:39.313455 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:39.318370 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:39.318346 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:40.042549 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:40.042523 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:40.043491 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:40.043475 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:40.526015 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:40.525989 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:45:40.609383 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:40.609353 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn"] Jul 22 09:45:40.612643 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:40.612629 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn" Jul 22 09:45:40.615047 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:40.615028 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-9e388426-serving-cert\"" Jul 22 09:45:40.615157 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:40.615140 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-876r5\"" Jul 22 09:45:40.623460 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:40.623436 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn"] Jul 22 09:45:40.633453 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:40.633432 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:45:40.733692 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:40.733662 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ac5a76-8fde-46d8-bfa0-323379d0c8a1-openshift-service-ca\") pod \"maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn\" (UID: \"30ac5a76-8fde-46d8-bfa0-323379d0c8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn" Jul 22 09:45:40.733692 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:40.733696 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh67d\" (UniqueName: \"kubernetes.io/projected/30ac5a76-8fde-46d8-bfa0-323379d0c8a1-kube-api-access-jh67d\") pod \"maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn\" (UID: \"30ac5a76-8fde-46d8-bfa0-323379d0c8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn" Jul 22 09:45:40.733899 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:40.733728 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ac5a76-8fde-46d8-bfa0-323379d0c8a1-maas-api-tls\") pod \"maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn\" (UID: \"30ac5a76-8fde-46d8-bfa0-323379d0c8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn" Jul 22 09:45:40.758777 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:40.758747 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:45:40.834445 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:40.834364 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ac5a76-8fde-46d8-bfa0-323379d0c8a1-openshift-service-ca\") pod \"maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn\" (UID: \"30ac5a76-8fde-46d8-bfa0-323379d0c8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn" Jul 22 09:45:40.834445 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:40.834404 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh67d\" (UniqueName: \"kubernetes.io/projected/30ac5a76-8fde-46d8-bfa0-323379d0c8a1-kube-api-access-jh67d\") pod \"maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn\" (UID: \"30ac5a76-8fde-46d8-bfa0-323379d0c8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn" Jul 22 09:45:40.834690 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:40.834530 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ac5a76-8fde-46d8-bfa0-323379d0c8a1-maas-api-tls\") pod \"maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn\" (UID: \"30ac5a76-8fde-46d8-bfa0-323379d0c8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn" Jul 22 09:45:40.835067 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:40.835045 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ac5a76-8fde-46d8-bfa0-323379d0c8a1-openshift-service-ca\") pod \"maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn\" (UID: \"30ac5a76-8fde-46d8-bfa0-323379d0c8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn" Jul 22 09:45:40.836842 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:40.836820 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ac5a76-8fde-46d8-bfa0-323379d0c8a1-maas-api-tls\") pod \"maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn\" (UID: \"30ac5a76-8fde-46d8-bfa0-323379d0c8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn" Jul 22 09:45:40.842001 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:40.841983 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh67d\" (UniqueName: \"kubernetes.io/projected/30ac5a76-8fde-46d8-bfa0-323379d0c8a1-kube-api-access-jh67d\") pod \"maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn\" (UID: \"30ac5a76-8fde-46d8-bfa0-323379d0c8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn" Jul 22 09:45:40.924453 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:40.924420 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn" Jul 22 09:45:41.056324 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:41.056275 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn"] Jul 22 09:45:41.059367 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:45:41.059343 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ac5a76_8fde_46d8_bfa0_323379d0c8a1.slice/crio-01696bd1a298faa601400444e6720d73319274588712ec4d18f409ccf04abce3 WatchSource:0}: Error finding container 01696bd1a298faa601400444e6720d73319274588712ec4d18f409ccf04abce3: Status 404 returned error can't find the container with id 01696bd1a298faa601400444e6720d73319274588712ec4d18f409ccf04abce3 Jul 22 09:45:41.118721 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:41.118641 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:45:41.219874 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:41.219839 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:45:42.055878 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.055843 2549 generic.go:356] "Generic (PLEG): container finished" podID="bd32224f-90ac-44d5-b492-39d31c6e5cad" containerID="165cc489ce26d251f6cf9680b89ba73e08bcf5ca89f08003e68499c487605cd2" exitCode=0 Jul 22 09:45:42.056094 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.056060 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" event={"ID":"bd32224f-90ac-44d5-b492-39d31c6e5cad","Type":"ContainerDied","Data":"165cc489ce26d251f6cf9680b89ba73e08bcf5ca89f08003e68499c487605cd2"} Jul 22 09:45:42.056283 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.056167 2549 scope.go:117] "RemoveContainer" containerID="6df4e5501d50d75cf81e0aeb3585cc5418a86f538ded9d0b5ee40efe94740a33" Jul 22 09:45:42.058772 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.058750 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn" event={"ID":"30ac5a76-8fde-46d8-bfa0-323379d0c8a1","Type":"ContainerStarted","Data":"45de76fc81bc172b4f8a700104c21e19ed19bcd01c4f92e88fa50faed0b6a157"} Jul 22 09:45:42.059171 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.058802 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn" event={"ID":"30ac5a76-8fde-46d8-bfa0-323379d0c8a1","Type":"ContainerStarted","Data":"01696bd1a298faa601400444e6720d73319274588712ec4d18f409ccf04abce3"} Jul 22 09:45:42.059232 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.059188 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn" Jul 22 09:45:42.085179 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.085127 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn" podStartSLOduration=1.772390874 podStartE2EDuration="2.085106935s" totalImagesPullingTime="312.716061ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:45:42.08339091 +0000 UTC m=+1672.547185350" watchObservedRunningTime="2026-07-22 09:45:42.085106935 +0000 UTC m=+1672.548901375" Jul 22 09:45:42.141391 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.141367 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:45:42.245978 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.245897 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-podinfo\") pod \"bd32224f-90ac-44d5-b492-39d31c6e5cad\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " Jul 22 09:45:42.245978 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.245939 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-token\") pod \"bd32224f-90ac-44d5-b492-39d31c6e5cad\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " Jul 22 09:45:42.245978 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.245967 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-data\") pod \"bd32224f-90ac-44d5-b492-39d31c6e5cad\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " Jul 22 09:45:42.246223 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.246087 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-envoy\") pod \"bd32224f-90ac-44d5-b492-39d31c6e5cad\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " Jul 22 09:45:42.246223 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.246133 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-credential-socket\") pod \"bd32224f-90ac-44d5-b492-39d31c6e5cad\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " Jul 22 09:45:42.246223 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.246175 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-workload-certs\") pod \"bd32224f-90ac-44d5-b492-39d31c6e5cad\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " Jul 22 09:45:42.246223 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.246199 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lglxf\" (UniqueName: \"kubernetes.io/projected/bd32224f-90ac-44d5-b492-39d31c6e5cad-kube-api-access-lglxf\") pod \"bd32224f-90ac-44d5-b492-39d31c6e5cad\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " Jul 22 09:45:42.246487 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.246297 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd32224f-90ac-44d5-b492-39d31c6e5cad-istiod-ca-cert\") pod \"bd32224f-90ac-44d5-b492-39d31c6e5cad\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " Jul 22 09:45:42.246487 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.246359 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-workload-socket\") pod \"bd32224f-90ac-44d5-b492-39d31c6e5cad\" (UID: \"bd32224f-90ac-44d5-b492-39d31c6e5cad\") " Jul 22 09:45:42.246487 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.246442 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bd32224f-90ac-44d5-b492-39d31c6e5cad" (UID: "bd32224f-90ac-44d5-b492-39d31c6e5cad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:45:42.246686 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.246663 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bd32224f-90ac-44d5-b492-39d31c6e5cad" (UID: "bd32224f-90ac-44d5-b492-39d31c6e5cad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:45:42.246751 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.246686 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:45:42.246807 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.246741 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bd32224f-90ac-44d5-b492-39d31c6e5cad" (UID: "bd32224f-90ac-44d5-b492-39d31c6e5cad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:45:42.247043 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.246980 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-data" (OuterVolumeSpecName: "istio-data") pod "bd32224f-90ac-44d5-b492-39d31c6e5cad" (UID: "bd32224f-90ac-44d5-b492-39d31c6e5cad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:45:42.247043 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.246983 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd32224f-90ac-44d5-b492-39d31c6e5cad-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bd32224f-90ac-44d5-b492-39d31c6e5cad" (UID: "bd32224f-90ac-44d5-b492-39d31c6e5cad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:45:42.248217 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.248191 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-token" (OuterVolumeSpecName: "istio-token") pod "bd32224f-90ac-44d5-b492-39d31c6e5cad" (UID: "bd32224f-90ac-44d5-b492-39d31c6e5cad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:45:42.248438 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.248216 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bd32224f-90ac-44d5-b492-39d31c6e5cad" (UID: "bd32224f-90ac-44d5-b492-39d31c6e5cad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 09:45:42.248438 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.248274 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bd32224f-90ac-44d5-b492-39d31c6e5cad" (UID: "bd32224f-90ac-44d5-b492-39d31c6e5cad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:45:42.248535 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.248443 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd32224f-90ac-44d5-b492-39d31c6e5cad-kube-api-access-lglxf" (OuterVolumeSpecName: "kube-api-access-lglxf") pod "bd32224f-90ac-44d5-b492-39d31c6e5cad" (UID: "bd32224f-90ac-44d5-b492-39d31c6e5cad"). InnerVolumeSpecName "kube-api-access-lglxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:45:42.347929 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.347908 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd32224f-90ac-44d5-b492-39d31c6e5cad-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:45:42.347929 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.347931 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:45:42.348055 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.347940 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:45:42.348055 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.347948 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:45:42.348055 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.347956 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:45:42.348055 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.347964 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:45:42.348055 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.347970 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd32224f-90ac-44d5-b492-39d31c6e5cad-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:45:42.348055 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:42.347978 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lglxf\" (UniqueName: \"kubernetes.io/projected/bd32224f-90ac-44d5-b492-39d31c6e5cad-kube-api-access-lglxf\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:45:43.065470 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:43.065441 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" Jul 22 09:45:43.066028 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:43.065493 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t" event={"ID":"bd32224f-90ac-44d5-b492-39d31c6e5cad","Type":"ContainerDied","Data":"eb748d3d9ef8932cb7c7a670dba6e6eaaf25e72b8e8f9fd26bfab3d2403950d8"} Jul 22 09:45:43.066028 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:43.065546 2549 scope.go:117] "RemoveContainer" containerID="165cc489ce26d251f6cf9680b89ba73e08bcf5ca89f08003e68499c487605cd2" Jul 22 09:45:43.086180 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:43.086157 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t"] Jul 22 09:45:43.091134 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:43.091112 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1d701482-openshift-default-6975647785-2wr5t"] Jul 22 09:45:44.340114 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:44.340081 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd32224f-90ac-44d5-b492-39d31c6e5cad" path="/var/lib/kubelet/pods/bd32224f-90ac-44d5-b492-39d31c6e5cad/volumes" Jul 22 09:45:49.085619 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:49.085584 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn" Jul 22 09:45:55.395511 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:55.394615 2549 patch_prober.go:28] interesting pod/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": read tcp 10.133.0.2:56348->10.133.0.39:15021: read: connection reset by peer" start-of-body= Jul 22 09:45:55.395511 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:55.394688 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" podUID="5de07f5e-bf70-4b82-825f-c7fa6bf6f945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": read tcp 10.133.0.2:56348->10.133.0.39:15021: read: connection reset by peer" Jul 22 09:45:56.130360 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:56.130324 2549 generic.go:356] "Generic (PLEG): container finished" podID="5de07f5e-bf70-4b82-825f-c7fa6bf6f945" containerID="c4df1ca3f3094c4d7a78bb57cdd1c064ad78b683df122ba6ea3fa86735c85142" exitCode=0 Jul 22 09:45:56.130360 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:56.130333 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" event={"ID":"5de07f5e-bf70-4b82-825f-c7fa6bf6f945","Type":"ContainerDied","Data":"c4df1ca3f3094c4d7a78bb57cdd1c064ad78b683df122ba6ea3fa86735c85142"} Jul 22 09:45:56.130727 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:56.130713 2549 scope.go:117] "RemoveContainer" containerID="c4df1ca3f3094c4d7a78bb57cdd1c064ad78b683df122ba6ea3fa86735c85142" Jul 22 09:45:56.131616 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:56.131588 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:45:56.131672 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:56.131655 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:45:56.131708 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:56.131687 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:45:57.136796 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:57.136760 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" event={"ID":"5de07f5e-bf70-4b82-825f-c7fa6bf6f945","Type":"ContainerStarted","Data":"bd93bd96b18680baad076ccf3d0a77e198688e296488a2836d03c9f393da56da"} Jul 22 09:45:57.313285 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:57.313245 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:45:57.313612 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:57.313590 2549 patch_prober.go:28] interesting pod/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 22 09:45:57.313659 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:57.313640 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" podUID="5de07f5e-bf70-4b82-825f-c7fa6bf6f945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 22 09:45:58.313805 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:58.313767 2549 patch_prober.go:28] interesting pod/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 22 09:45:58.314208 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:58.313823 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" podUID="5de07f5e-bf70-4b82-825f-c7fa6bf6f945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 22 09:45:59.314446 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:59.314409 2549 patch_prober.go:28] interesting pod/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 22 09:45:59.314832 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:45:59.314489 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" podUID="5de07f5e-bf70-4b82-825f-c7fa6bf6f945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 22 09:46:00.314057 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:00.314018 2549 patch_prober.go:28] interesting pod/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 22 09:46:00.314234 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:00.314081 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" podUID="5de07f5e-bf70-4b82-825f-c7fa6bf6f945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 22 09:46:01.317053 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:01.317024 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:46:01.317463 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:01.317317 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:46:01.317927 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:01.317910 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:46:16.043276 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:16.043201 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn"] Jul 22 09:46:16.043785 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:16.043426 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn" podUID="30ac5a76-8fde-46d8-bfa0-323379d0c8a1" containerName="maas-api" containerID="cri-o://45de76fc81bc172b4f8a700104c21e19ed19bcd01c4f92e88fa50faed0b6a157" gracePeriod=30 Jul 22 09:46:16.233502 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:16.233466 2549 generic.go:356] "Generic (PLEG): container finished" podID="30ac5a76-8fde-46d8-bfa0-323379d0c8a1" containerID="45de76fc81bc172b4f8a700104c21e19ed19bcd01c4f92e88fa50faed0b6a157" exitCode=0 Jul 22 09:46:16.233665 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:16.233520 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn" event={"ID":"30ac5a76-8fde-46d8-bfa0-323379d0c8a1","Type":"ContainerDied","Data":"45de76fc81bc172b4f8a700104c21e19ed19bcd01c4f92e88fa50faed0b6a157"} Jul 22 09:46:16.263369 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:16.263298 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:46:16.293335 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:16.293263 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn" Jul 22 09:46:16.350141 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:16.350116 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ac5a76-8fde-46d8-bfa0-323379d0c8a1-openshift-service-ca\") pod \"30ac5a76-8fde-46d8-bfa0-323379d0c8a1\" (UID: \"30ac5a76-8fde-46d8-bfa0-323379d0c8a1\") " Jul 22 09:46:16.350290 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:16.350146 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ac5a76-8fde-46d8-bfa0-323379d0c8a1-maas-api-tls\") pod \"30ac5a76-8fde-46d8-bfa0-323379d0c8a1\" (UID: \"30ac5a76-8fde-46d8-bfa0-323379d0c8a1\") " Jul 22 09:46:16.350290 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:16.350193 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh67d\" (UniqueName: \"kubernetes.io/projected/30ac5a76-8fde-46d8-bfa0-323379d0c8a1-kube-api-access-jh67d\") pod \"30ac5a76-8fde-46d8-bfa0-323379d0c8a1\" (UID: \"30ac5a76-8fde-46d8-bfa0-323379d0c8a1\") " Jul 22 09:46:16.350494 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:16.350470 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ac5a76-8fde-46d8-bfa0-323379d0c8a1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "30ac5a76-8fde-46d8-bfa0-323379d0c8a1" (UID: "30ac5a76-8fde-46d8-bfa0-323379d0c8a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:46:16.352146 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:16.352123 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ac5a76-8fde-46d8-bfa0-323379d0c8a1-kube-api-access-jh67d" (OuterVolumeSpecName: "kube-api-access-jh67d") pod "30ac5a76-8fde-46d8-bfa0-323379d0c8a1" (UID: "30ac5a76-8fde-46d8-bfa0-323379d0c8a1"). InnerVolumeSpecName "kube-api-access-jh67d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:46:16.352231 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:16.352162 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ac5a76-8fde-46d8-bfa0-323379d0c8a1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "30ac5a76-8fde-46d8-bfa0-323379d0c8a1" (UID: "30ac5a76-8fde-46d8-bfa0-323379d0c8a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:46:16.450751 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:16.450717 2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ac5a76-8fde-46d8-bfa0-323379d0c8a1-openshift-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:46:16.450751 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:16.450746 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ac5a76-8fde-46d8-bfa0-323379d0c8a1-maas-api-tls\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:46:16.450751 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:16.450755 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh67d\" (UniqueName: \"kubernetes.io/projected/30ac5a76-8fde-46d8-bfa0-323379d0c8a1-kube-api-access-jh67d\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:46:17.240082 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:17.240047 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn" event={"ID":"30ac5a76-8fde-46d8-bfa0-323379d0c8a1","Type":"ContainerDied","Data":"01696bd1a298faa601400444e6720d73319274588712ec4d18f409ccf04abce3"} Jul 22 09:46:17.240541 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:17.240103 2549 scope.go:117] "RemoveContainer" containerID="45de76fc81bc172b4f8a700104c21e19ed19bcd01c4f92e88fa50faed0b6a157" Jul 22 09:46:17.240541 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:17.240059 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn" Jul 22 09:46:17.267072 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:17.267049 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn"] Jul 22 09:46:17.276769 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:17.276748 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e388426-6b9c5664f8-hjdrn"] Jul 22 09:46:18.340062 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:18.340031 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30ac5a76-8fde-46d8-bfa0-323379d0c8a1" path="/var/lib/kubelet/pods/30ac5a76-8fde-46d8-bfa0-323379d0c8a1/volumes" Jul 22 09:46:20.557543 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:20.557513 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:46:20.688189 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:20.688157 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r"] Jul 22 09:46:20.688453 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:20.688424 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" podUID="5de07f5e-bf70-4b82-825f-c7fa6bf6f945" containerName="istio-proxy" containerID="cri-o://bd93bd96b18680baad076ccf3d0a77e198688e296488a2836d03c9f393da56da" gracePeriod=30 Jul 22 09:46:20.713556 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:20.713525 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:46:20.936077 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:20.935993 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:46:21.166768 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.166734 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:46:21.451864 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.451826 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7"] Jul 22 09:46:21.452138 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.452127 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30ac5a76-8fde-46d8-bfa0-323379d0c8a1" containerName="maas-api" Jul 22 09:46:21.452182 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.452140 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="30ac5a76-8fde-46d8-bfa0-323379d0c8a1" containerName="maas-api" Jul 22 09:46:21.452182 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.452153 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd32224f-90ac-44d5-b492-39d31c6e5cad" containerName="istio-proxy" Jul 22 09:46:21.452182 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.452159 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd32224f-90ac-44d5-b492-39d31c6e5cad" containerName="istio-proxy" Jul 22 09:46:21.452182 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.452169 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd32224f-90ac-44d5-b492-39d31c6e5cad" containerName="istio-proxy" Jul 22 09:46:21.452182 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.452175 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd32224f-90ac-44d5-b492-39d31c6e5cad" containerName="istio-proxy" Jul 22 09:46:21.452360 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.452225 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd32224f-90ac-44d5-b492-39d31c6e5cad" containerName="istio-proxy" Jul 22 09:46:21.452360 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.452232 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="30ac5a76-8fde-46d8-bfa0-323379d0c8a1" containerName="maas-api" Jul 22 09:46:21.452360 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.452239 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd32224f-90ac-44d5-b492-39d31c6e5cad" containerName="istio-proxy" Jul 22 09:46:21.455088 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.455073 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.457413 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.457386 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-424990ec-openshift-default-dockercfg-tzllr\"" Jul 22 09:46:21.468035 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.468014 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7"] Jul 22 09:46:21.595239 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.595206 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-istio-data\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.595239 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.595240 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a25d30-e365-4d20-8367-469fb10e36a7-istiod-ca-cert\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.595668 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.595287 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-workload-certs\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.595668 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.595331 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-istio-envoy\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.595668 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.595384 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l4w5\" (UniqueName: \"kubernetes.io/projected/25a25d30-e365-4d20-8367-469fb10e36a7-kube-api-access-8l4w5\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.595668 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.595402 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-credential-socket\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.595668 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.595437 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-workload-socket\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.595668 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.595463 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a25d30-e365-4d20-8367-469fb10e36a7-istio-podinfo\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.595668 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.595488 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a25d30-e365-4d20-8367-469fb10e36a7-istio-token\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.696115 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.696075 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l4w5\" (UniqueName: \"kubernetes.io/projected/25a25d30-e365-4d20-8367-469fb10e36a7-kube-api-access-8l4w5\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.696115 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.696115 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-credential-socket\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.696422 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.696144 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-workload-socket\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.696422 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.696269 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a25d30-e365-4d20-8367-469fb10e36a7-istio-podinfo\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.696422 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.696342 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a25d30-e365-4d20-8367-469fb10e36a7-istio-token\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.696584 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.696427 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-istio-data\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.696584 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.696454 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a25d30-e365-4d20-8367-469fb10e36a7-istiod-ca-cert\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.696584 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.696519 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-workload-certs\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.696584 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.696546 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-istio-envoy\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.696584 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.696554 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-credential-socket\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.696837 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.696581 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-workload-socket\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.696837 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.696767 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-istio-data\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.696837 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.696769 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-workload-certs\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.697173 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.697154 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a25d30-e365-4d20-8367-469fb10e36a7-istiod-ca-cert\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.698648 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.698629 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-istio-envoy\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.698875 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.698859 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a25d30-e365-4d20-8367-469fb10e36a7-istio-podinfo\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.705651 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.705596 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l4w5\" (UniqueName: \"kubernetes.io/projected/25a25d30-e365-4d20-8367-469fb10e36a7-kube-api-access-8l4w5\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.705990 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.705970 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a25d30-e365-4d20-8367-469fb10e36a7-istio-token\") pod \"e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.764912 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.764884 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:21.908966 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.908940 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7"] Jul 22 09:46:21.910556 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:46:21.910530 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a25d30_e365_4d20_8367_469fb10e36a7.slice/crio-5a48aa698aa6514cfe08cfd5a93ba987a5464a736399af32739f4300afcf79bf WatchSource:0}: Error finding container 5a48aa698aa6514cfe08cfd5a93ba987a5464a736399af32739f4300afcf79bf: Status 404 returned error can't find the container with id 5a48aa698aa6514cfe08cfd5a93ba987a5464a736399af32739f4300afcf79bf Jul 22 09:46:21.912692 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.912653 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:46:21.912772 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.912735 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:46:21.912819 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:21.912777 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:46:22.266673 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:22.266579 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" event={"ID":"25a25d30-e365-4d20-8367-469fb10e36a7","Type":"ContainerStarted","Data":"0d4005b50a4f89c997939ee633decd7e5eac4b1436db0578ed1f1c4a9950914f"} Jul 22 09:46:22.266673 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:22.266648 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" event={"ID":"25a25d30-e365-4d20-8367-469fb10e36a7","Type":"ContainerStarted","Data":"5a48aa698aa6514cfe08cfd5a93ba987a5464a736399af32739f4300afcf79bf"} Jul 22 09:46:22.290113 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:22.290065 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" podStartSLOduration=1.290048636 podStartE2EDuration="1.290048636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:46:22.288557035 +0000 UTC m=+1712.752351472" watchObservedRunningTime="2026-07-22 09:46:22.290048636 +0000 UTC m=+1712.753843074" Jul 22 09:46:22.482626 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:22.482592 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:46:22.582623 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:22.582593 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:46:22.765375 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:22.765343 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:22.766652 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:22.766628 2549 patch_prober.go:28] interesting pod/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 22 09:46:22.766774 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:22.766684 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" podUID="25a25d30-e365-4d20-8367-469fb10e36a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 22 09:46:23.615975 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:23.615943 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:46:23.706160 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:23.706128 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt"] Jul 22 09:46:23.709506 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:23.709484 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt" Jul 22 09:46:23.713008 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:23.712951 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-876r5\"" Jul 22 09:46:23.713008 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:23.712966 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-424990ec-serving-cert\"" Jul 22 09:46:23.721963 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:23.721937 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt"] Jul 22 09:46:23.734905 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:23.734880 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:46:23.769413 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:23.769385 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:23.817586 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:23.817551 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95961d70-5d64-47c3-8998-9e98e7fb3e4e-openshift-service-ca\") pod \"maas-api-e2e-ait-424990ec-65657476f6-dbslt\" (UID: \"95961d70-5d64-47c3-8998-9e98e7fb3e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt" Jul 22 09:46:23.817746 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:23.817610 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9g2x\" (UniqueName: \"kubernetes.io/projected/95961d70-5d64-47c3-8998-9e98e7fb3e4e-kube-api-access-d9g2x\") pod \"maas-api-e2e-ait-424990ec-65657476f6-dbslt\" (UID: \"95961d70-5d64-47c3-8998-9e98e7fb3e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt" Jul 22 09:46:23.817746 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:23.817649 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95961d70-5d64-47c3-8998-9e98e7fb3e4e-maas-api-tls\") pod \"maas-api-e2e-ait-424990ec-65657476f6-dbslt\" (UID: \"95961d70-5d64-47c3-8998-9e98e7fb3e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt" Jul 22 09:46:23.836702 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:23.836671 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:46:23.918742 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:23.918656 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95961d70-5d64-47c3-8998-9e98e7fb3e4e-openshift-service-ca\") pod \"maas-api-e2e-ait-424990ec-65657476f6-dbslt\" (UID: \"95961d70-5d64-47c3-8998-9e98e7fb3e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt" Jul 22 09:46:23.918742 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:23.918702 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9g2x\" (UniqueName: \"kubernetes.io/projected/95961d70-5d64-47c3-8998-9e98e7fb3e4e-kube-api-access-d9g2x\") pod \"maas-api-e2e-ait-424990ec-65657476f6-dbslt\" (UID: \"95961d70-5d64-47c3-8998-9e98e7fb3e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt" Jul 22 09:46:23.918742 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:23.918727 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95961d70-5d64-47c3-8998-9e98e7fb3e4e-maas-api-tls\") pod \"maas-api-e2e-ait-424990ec-65657476f6-dbslt\" (UID: \"95961d70-5d64-47c3-8998-9e98e7fb3e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt" Jul 22 09:46:23.919021 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:46:23.918830 2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-serving-cert: secret "maas-api-e2e-ait-424990ec-serving-cert" not found Jul 22 09:46:23.919021 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:46:23.918887 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95961d70-5d64-47c3-8998-9e98e7fb3e4e-maas-api-tls podName:95961d70-5d64-47c3-8998-9e98e7fb3e4e nodeName:}" failed. No retries permitted until 2026-07-22 09:46:24.41886917 +0000 UTC m=+1714.882663586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/95961d70-5d64-47c3-8998-9e98e7fb3e4e-maas-api-tls") pod "maas-api-e2e-ait-424990ec-65657476f6-dbslt" (UID: "95961d70-5d64-47c3-8998-9e98e7fb3e4e") : secret "maas-api-e2e-ait-424990ec-serving-cert" not found Jul 22 09:46:23.919433 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:23.919409 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95961d70-5d64-47c3-8998-9e98e7fb3e4e-openshift-service-ca\") pod \"maas-api-e2e-ait-424990ec-65657476f6-dbslt\" (UID: \"95961d70-5d64-47c3-8998-9e98e7fb3e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt" Jul 22 09:46:23.930660 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:23.930635 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9g2x\" (UniqueName: \"kubernetes.io/projected/95961d70-5d64-47c3-8998-9e98e7fb3e4e-kube-api-access-d9g2x\") pod \"maas-api-e2e-ait-424990ec-65657476f6-dbslt\" (UID: \"95961d70-5d64-47c3-8998-9e98e7fb3e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt" Jul 22 09:46:24.004079 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:24.004046 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:46:24.275898 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:24.275869 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:24.276785 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:24.276765 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:24.423598 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:24.423558 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95961d70-5d64-47c3-8998-9e98e7fb3e4e-maas-api-tls\") pod \"maas-api-e2e-ait-424990ec-65657476f6-dbslt\" (UID: \"95961d70-5d64-47c3-8998-9e98e7fb3e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt" Jul 22 09:46:24.426013 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:24.425987 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95961d70-5d64-47c3-8998-9e98e7fb3e4e-maas-api-tls\") pod \"maas-api-e2e-ait-424990ec-65657476f6-dbslt\" (UID: \"95961d70-5d64-47c3-8998-9e98e7fb3e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt" Jul 22 09:46:24.449821 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:24.449794 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:46:24.553981 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:24.553894 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:46:24.622495 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:24.622459 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt" Jul 22 09:46:24.752639 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:24.752614 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt"] Jul 22 09:46:24.754620 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:46:24.754593 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95961d70_5d64_47c3_8998_9e98e7fb3e4e.slice/crio-c3c0e8af2b876c8f78b18f5bf76dd160d4c6b5610c1d05141baf29dabc98c11e WatchSource:0}: Error finding container c3c0e8af2b876c8f78b18f5bf76dd160d4c6b5610c1d05141baf29dabc98c11e: Status 404 returned error can't find the container with id c3c0e8af2b876c8f78b18f5bf76dd160d4c6b5610c1d05141baf29dabc98c11e Jul 22 09:46:25.281617 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:25.281585 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt" event={"ID":"95961d70-5d64-47c3-8998-9e98e7fb3e4e","Type":"ContainerStarted","Data":"c3c0e8af2b876c8f78b18f5bf76dd160d4c6b5610c1d05141baf29dabc98c11e"} Jul 22 09:46:25.938125 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:25.938101 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:46:26.039032 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.038999 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-token\") pod \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " Jul 22 09:46:26.039220 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.039058 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-data\") pod \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " Jul 22 09:46:26.039220 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.039086 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-podinfo\") pod \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " Jul 22 09:46:26.039220 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.039112 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-envoy\") pod \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " Jul 22 09:46:26.039220 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.039159 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-workload-certs\") pod \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " Jul 22 09:46:26.039220 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.039204 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-credential-socket\") pod \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " Jul 22 09:46:26.039556 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.039251 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-workload-socket\") pod \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " Jul 22 09:46:26.039556 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.039290 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istiod-ca-cert\") pod \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " Jul 22 09:46:26.039556 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.039351 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f92rc\" (UniqueName: \"kubernetes.io/projected/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-kube-api-access-f92rc\") pod \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\" (UID: \"5de07f5e-bf70-4b82-825f-c7fa6bf6f945\") " Jul 22 09:46:26.039697 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.039651 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5de07f5e-bf70-4b82-825f-c7fa6bf6f945" (UID: "5de07f5e-bf70-4b82-825f-c7fa6bf6f945"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:46:26.039803 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.039775 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5de07f5e-bf70-4b82-825f-c7fa6bf6f945" (UID: "5de07f5e-bf70-4b82-825f-c7fa6bf6f945"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:46:26.039873 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.039799 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5de07f5e-bf70-4b82-825f-c7fa6bf6f945" (UID: "5de07f5e-bf70-4b82-825f-c7fa6bf6f945"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:46:26.039873 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.039853 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5de07f5e-bf70-4b82-825f-c7fa6bf6f945" (UID: "5de07f5e-bf70-4b82-825f-c7fa6bf6f945"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:46:26.039970 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.039870 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-data" (OuterVolumeSpecName: "istio-data") pod "5de07f5e-bf70-4b82-825f-c7fa6bf6f945" (UID: "5de07f5e-bf70-4b82-825f-c7fa6bf6f945"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:46:26.041337 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.041315 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5de07f5e-bf70-4b82-825f-c7fa6bf6f945" (UID: "5de07f5e-bf70-4b82-825f-c7fa6bf6f945"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 09:46:26.041695 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.041668 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-kube-api-access-f92rc" (OuterVolumeSpecName: "kube-api-access-f92rc") pod "5de07f5e-bf70-4b82-825f-c7fa6bf6f945" (UID: "5de07f5e-bf70-4b82-825f-c7fa6bf6f945"). InnerVolumeSpecName "kube-api-access-f92rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:46:26.041802 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.041724 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5de07f5e-bf70-4b82-825f-c7fa6bf6f945" (UID: "5de07f5e-bf70-4b82-825f-c7fa6bf6f945"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:46:26.041802 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.041759 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-token" (OuterVolumeSpecName: "istio-token") pod "5de07f5e-bf70-4b82-825f-c7fa6bf6f945" (UID: "5de07f5e-bf70-4b82-825f-c7fa6bf6f945"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:46:26.141039 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.141002 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:46:26.141039 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.141032 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:46:26.141039 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.141044 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f92rc\" (UniqueName: \"kubernetes.io/projected/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-kube-api-access-f92rc\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:46:26.141277 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.141054 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:46:26.141277 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.141063 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:46:26.141277 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.141070 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:46:26.141277 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.141077 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:46:26.141277 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.141085 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:46:26.141277 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.141092 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de07f5e-bf70-4b82-825f-c7fa6bf6f945-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:46:26.289102 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.289005 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt" event={"ID":"95961d70-5d64-47c3-8998-9e98e7fb3e4e","Type":"ContainerStarted","Data":"ec235735ccca13398ca9671f481623ee02eceaf235705c111feb878a70ceb65e"} Jul 22 09:46:26.289102 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.289078 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt" Jul 22 09:46:26.290692 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.290664 2549 generic.go:356] "Generic (PLEG): container finished" podID="5de07f5e-bf70-4b82-825f-c7fa6bf6f945" containerID="bd93bd96b18680baad076ccf3d0a77e198688e296488a2836d03c9f393da56da" exitCode=0 Jul 22 09:46:26.290783 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.290700 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" event={"ID":"5de07f5e-bf70-4b82-825f-c7fa6bf6f945","Type":"ContainerDied","Data":"bd93bd96b18680baad076ccf3d0a77e198688e296488a2836d03c9f393da56da"} Jul 22 09:46:26.290783 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.290747 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" event={"ID":"5de07f5e-bf70-4b82-825f-c7fa6bf6f945","Type":"ContainerDied","Data":"0b2595c8344954c960a4dab1a84ac86772de351182449dcef732289cf1f76a20"} Jul 22 09:46:26.290783 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.290771 2549 scope.go:117] "RemoveContainer" containerID="bd93bd96b18680baad076ccf3d0a77e198688e296488a2836d03c9f393da56da" Jul 22 09:46:26.290909 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.290707 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r" Jul 22 09:46:26.300854 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.300836 2549 scope.go:117] "RemoveContainer" containerID="c4df1ca3f3094c4d7a78bb57cdd1c064ad78b683df122ba6ea3fa86735c85142" Jul 22 09:46:26.307750 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.307705 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt" podStartSLOduration=2.761718688 podStartE2EDuration="3.307688631s" totalImagesPullingTime="545.969943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:46:26.307452974 +0000 UTC m=+1716.771247495" watchObservedRunningTime="2026-07-22 09:46:26.307688631 +0000 UTC m=+1716.771483069" Jul 22 09:46:26.308791 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.308767 2549 scope.go:117] "RemoveContainer" containerID="bd93bd96b18680baad076ccf3d0a77e198688e296488a2836d03c9f393da56da" Jul 22 09:46:26.309024 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:46:26.309003 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd93bd96b18680baad076ccf3d0a77e198688e296488a2836d03c9f393da56da\": container with ID starting with bd93bd96b18680baad076ccf3d0a77e198688e296488a2836d03c9f393da56da not found: ID does not exist" containerID="bd93bd96b18680baad076ccf3d0a77e198688e296488a2836d03c9f393da56da" Jul 22 09:46:26.309076 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.309033 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd93bd96b18680baad076ccf3d0a77e198688e296488a2836d03c9f393da56da"} err="failed to get container status \"bd93bd96b18680baad076ccf3d0a77e198688e296488a2836d03c9f393da56da\": rpc error: code = NotFound desc = could not find container \"bd93bd96b18680baad076ccf3d0a77e198688e296488a2836d03c9f393da56da\": container with ID starting with bd93bd96b18680baad076ccf3d0a77e198688e296488a2836d03c9f393da56da not found: ID does not exist" Jul 22 09:46:26.309076 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.309059 2549 scope.go:117] "RemoveContainer" containerID="c4df1ca3f3094c4d7a78bb57cdd1c064ad78b683df122ba6ea3fa86735c85142" Jul 22 09:46:26.309289 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:46:26.309272 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4df1ca3f3094c4d7a78bb57cdd1c064ad78b683df122ba6ea3fa86735c85142\": container with ID starting with c4df1ca3f3094c4d7a78bb57cdd1c064ad78b683df122ba6ea3fa86735c85142 not found: ID does not exist" containerID="c4df1ca3f3094c4d7a78bb57cdd1c064ad78b683df122ba6ea3fa86735c85142" Jul 22 09:46:26.309357 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.309295 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4df1ca3f3094c4d7a78bb57cdd1c064ad78b683df122ba6ea3fa86735c85142"} err="failed to get container status \"c4df1ca3f3094c4d7a78bb57cdd1c064ad78b683df122ba6ea3fa86735c85142\": rpc error: code = NotFound desc = could not find container \"c4df1ca3f3094c4d7a78bb57cdd1c064ad78b683df122ba6ea3fa86735c85142\": container with ID starting with c4df1ca3f3094c4d7a78bb57cdd1c064ad78b683df122ba6ea3fa86735c85142 not found: ID does not exist" Jul 22 09:46:26.325482 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.325460 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r"] Jul 22 09:46:26.333410 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.333391 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-9e388426-gw-openshift-default-6d6c86d77f-d6c5r"] Jul 22 09:46:26.339194 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:26.339175 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5de07f5e-bf70-4b82-825f-c7fa6bf6f945" path="/var/lib/kubelet/pods/5de07f5e-bf70-4b82-825f-c7fa6bf6f945/volumes" Jul 22 09:46:32.302289 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:32.302252 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt" Jul 22 09:46:39.277629 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:39.277597 2549 patch_prober.go:28] interesting pod/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 22 09:46:39.278054 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:39.277652 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" podUID="25a25d30-e365-4d20-8367-469fb10e36a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 22 09:46:39.355650 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:39.355619 2549 generic.go:356] "Generic (PLEG): container finished" podID="25a25d30-e365-4d20-8367-469fb10e36a7" containerID="0d4005b50a4f89c997939ee633decd7e5eac4b1436db0578ed1f1c4a9950914f" exitCode=0 Jul 22 09:46:39.355837 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:39.355695 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" event={"ID":"25a25d30-e365-4d20-8367-469fb10e36a7","Type":"ContainerDied","Data":"0d4005b50a4f89c997939ee633decd7e5eac4b1436db0578ed1f1c4a9950914f"} Jul 22 09:46:39.356119 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:39.356102 2549 scope.go:117] "RemoveContainer" containerID="0d4005b50a4f89c997939ee633decd7e5eac4b1436db0578ed1f1c4a9950914f" Jul 22 09:46:39.357059 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:39.357031 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:46:39.357140 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:39.357101 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:46:39.357140 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:39.357133 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:46:40.362387 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:40.362350 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" event={"ID":"25a25d30-e365-4d20-8367-469fb10e36a7","Type":"ContainerStarted","Data":"9341e27958a874a8fa31f40d1ce40ae6715a1a791ab2e93c00a834d81629d354"} Jul 22 09:46:40.765229 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:40.765131 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:40.765439 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:40.765378 2549 patch_prober.go:28] interesting pod/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 22 09:46:40.765439 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:40.765425 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" podUID="25a25d30-e365-4d20-8367-469fb10e36a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 22 09:46:41.765286 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:41.765245 2549 patch_prober.go:28] interesting pod/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 22 09:46:41.765789 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:41.765328 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" podUID="25a25d30-e365-4d20-8367-469fb10e36a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 22 09:46:42.765543 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:42.765505 2549 patch_prober.go:28] interesting pod/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 22 09:46:42.766157 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:42.766122 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" podUID="25a25d30-e365-4d20-8367-469fb10e36a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 22 09:46:43.765594 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:43.765546 2549 patch_prober.go:28] interesting pod/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 22 09:46:43.766118 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:43.765618 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" podUID="25a25d30-e365-4d20-8367-469fb10e36a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 22 09:46:44.769108 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:44.769080 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:44.769565 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:44.769336 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:44.770061 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:44.770045 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:46:59.957566 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:59.957523 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt"] Jul 22 09:46:59.958013 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:59.957746 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt" podUID="95961d70-5d64-47c3-8998-9e98e7fb3e4e" containerName="maas-api" containerID="cri-o://ec235735ccca13398ca9671f481623ee02eceaf235705c111feb878a70ceb65e" gracePeriod=30 Jul 22 09:46:59.988345 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:46:59.988240 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:00.208411 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:00.208350 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt" Jul 22 09:47:00.338704 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:00.337873 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9g2x\" (UniqueName: \"kubernetes.io/projected/95961d70-5d64-47c3-8998-9e98e7fb3e4e-kube-api-access-d9g2x\") pod \"95961d70-5d64-47c3-8998-9e98e7fb3e4e\" (UID: \"95961d70-5d64-47c3-8998-9e98e7fb3e4e\") " Jul 22 09:47:00.338704 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:00.337961 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95961d70-5d64-47c3-8998-9e98e7fb3e4e-openshift-service-ca\") pod \"95961d70-5d64-47c3-8998-9e98e7fb3e4e\" (UID: \"95961d70-5d64-47c3-8998-9e98e7fb3e4e\") " Jul 22 09:47:00.338704 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:00.338002 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95961d70-5d64-47c3-8998-9e98e7fb3e4e-maas-api-tls\") pod \"95961d70-5d64-47c3-8998-9e98e7fb3e4e\" (UID: \"95961d70-5d64-47c3-8998-9e98e7fb3e4e\") " Jul 22 09:47:00.339657 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:00.339619 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95961d70-5d64-47c3-8998-9e98e7fb3e4e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "95961d70-5d64-47c3-8998-9e98e7fb3e4e" (UID: "95961d70-5d64-47c3-8998-9e98e7fb3e4e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:47:00.341426 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:00.341352 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95961d70-5d64-47c3-8998-9e98e7fb3e4e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "95961d70-5d64-47c3-8998-9e98e7fb3e4e" (UID: "95961d70-5d64-47c3-8998-9e98e7fb3e4e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:47:00.344683 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:00.344658 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95961d70-5d64-47c3-8998-9e98e7fb3e4e-kube-api-access-d9g2x" (OuterVolumeSpecName: "kube-api-access-d9g2x") pod "95961d70-5d64-47c3-8998-9e98e7fb3e4e" (UID: "95961d70-5d64-47c3-8998-9e98e7fb3e4e"). InnerVolumeSpecName "kube-api-access-d9g2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:47:00.439487 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:00.439450 2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95961d70-5d64-47c3-8998-9e98e7fb3e4e-openshift-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:00.439487 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:00.439482 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95961d70-5d64-47c3-8998-9e98e7fb3e4e-maas-api-tls\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:00.439487 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:00.439491 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9g2x\" (UniqueName: \"kubernetes.io/projected/95961d70-5d64-47c3-8998-9e98e7fb3e4e-kube-api-access-d9g2x\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:00.472217 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:00.472138 2549 generic.go:356] "Generic (PLEG): container finished" podID="95961d70-5d64-47c3-8998-9e98e7fb3e4e" containerID="ec235735ccca13398ca9671f481623ee02eceaf235705c111feb878a70ceb65e" exitCode=0 Jul 22 09:47:00.472217 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:00.472197 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt" Jul 22 09:47:00.472434 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:00.472227 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt" event={"ID":"95961d70-5d64-47c3-8998-9e98e7fb3e4e","Type":"ContainerDied","Data":"ec235735ccca13398ca9671f481623ee02eceaf235705c111feb878a70ceb65e"} Jul 22 09:47:00.472434 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:00.472291 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt" event={"ID":"95961d70-5d64-47c3-8998-9e98e7fb3e4e","Type":"ContainerDied","Data":"c3c0e8af2b876c8f78b18f5bf76dd160d4c6b5610c1d05141baf29dabc98c11e"} Jul 22 09:47:00.472434 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:00.472328 2549 scope.go:117] "RemoveContainer" containerID="ec235735ccca13398ca9671f481623ee02eceaf235705c111feb878a70ceb65e" Jul 22 09:47:00.480862 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:00.480846 2549 scope.go:117] "RemoveContainer" containerID="ec235735ccca13398ca9671f481623ee02eceaf235705c111feb878a70ceb65e" Jul 22 09:47:00.481108 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:47:00.481090 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec235735ccca13398ca9671f481623ee02eceaf235705c111feb878a70ceb65e\": container with ID starting with ec235735ccca13398ca9671f481623ee02eceaf235705c111feb878a70ceb65e not found: ID does not exist" containerID="ec235735ccca13398ca9671f481623ee02eceaf235705c111feb878a70ceb65e" Jul 22 09:47:00.481168 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:00.481119 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec235735ccca13398ca9671f481623ee02eceaf235705c111feb878a70ceb65e"} err="failed to get container status \"ec235735ccca13398ca9671f481623ee02eceaf235705c111feb878a70ceb65e\": rpc error: code = NotFound desc = could not find container \"ec235735ccca13398ca9671f481623ee02eceaf235705c111feb878a70ceb65e\": container with ID starting with ec235735ccca13398ca9671f481623ee02eceaf235705c111feb878a70ceb65e not found: ID does not exist" Jul 22 09:47:00.493611 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:00.493577 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt"] Jul 22 09:47:00.499084 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:00.499060 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-424990ec-65657476f6-dbslt"] Jul 22 09:47:02.339460 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:02.339428 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95961d70-5d64-47c3-8998-9e98e7fb3e4e" path="/var/lib/kubelet/pods/95961d70-5d64-47c3-8998-9e98e7fb3e4e/volumes" Jul 22 09:47:04.499337 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:04.499290 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:05.614351 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:05.614294 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7"] Jul 22 09:47:05.614765 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:05.614535 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" podUID="25a25d30-e365-4d20-8367-469fb10e36a7" containerName="istio-proxy" containerID="cri-o://9341e27958a874a8fa31f40d1ce40ae6715a1a791ab2e93c00a834d81629d354" gracePeriod=30 Jul 22 09:47:05.641566 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:05.641537 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:05.849829 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:05.849796 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:06.086860 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.086826 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:06.362348 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.362243 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q"] Jul 22 09:47:06.362575 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.362561 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95961d70-5d64-47c3-8998-9e98e7fb3e4e" containerName="maas-api" Jul 22 09:47:06.362575 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.362575 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="95961d70-5d64-47c3-8998-9e98e7fb3e4e" containerName="maas-api" Jul 22 09:47:06.362699 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.362592 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5de07f5e-bf70-4b82-825f-c7fa6bf6f945" containerName="istio-proxy" Jul 22 09:47:06.362699 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.362598 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="5de07f5e-bf70-4b82-825f-c7fa6bf6f945" containerName="istio-proxy" Jul 22 09:47:06.362699 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.362610 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5de07f5e-bf70-4b82-825f-c7fa6bf6f945" containerName="istio-proxy" Jul 22 09:47:06.362699 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.362615 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="5de07f5e-bf70-4b82-825f-c7fa6bf6f945" containerName="istio-proxy" Jul 22 09:47:06.362699 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.362661 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="5de07f5e-bf70-4b82-825f-c7fa6bf6f945" containerName="istio-proxy" Jul 22 09:47:06.362699 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.362669 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="95961d70-5d64-47c3-8998-9e98e7fb3e4e" containerName="maas-api" Jul 22 09:47:06.366973 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.366953 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.369497 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.369475 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-d055000b-openshift-default-dockercfg-frm4c\"" Jul 22 09:47:06.378764 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.378732 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q"] Jul 22 09:47:06.391898 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.391875 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:06.489273 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.489241 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-data\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.489460 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.489290 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-envoy\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.489460 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.489344 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-token\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.489460 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.489374 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istiod-ca-cert\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.489460 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.489410 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-podinfo\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.489460 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.489454 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-workload-socket\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.489636 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.489476 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-workload-certs\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.489636 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.489494 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-credential-socket\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.489636 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.489566 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqnsg\" (UniqueName: \"kubernetes.io/projected/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-kube-api-access-zqnsg\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.590432 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.590394 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-data\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.590606 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.590454 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-envoy\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.590606 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.590479 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-token\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.590606 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.590502 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istiod-ca-cert\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.590606 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.590531 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-podinfo\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.590815 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.590669 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-workload-socket\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.590815 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.590723 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-workload-certs\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.590815 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.590752 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-credential-socket\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.590815 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.590798 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqnsg\" (UniqueName: \"kubernetes.io/projected/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-kube-api-access-zqnsg\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.591025 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.590864 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-data\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.591079 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.591053 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-workload-socket\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.591269 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.591249 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istiod-ca-cert\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.591386 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.591344 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-workload-certs\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.591386 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.591367 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-credential-socket\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.592826 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.592804 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-envoy\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.593045 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.593025 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-podinfo\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.607512 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.607484 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqnsg\" (UniqueName: \"kubernetes.io/projected/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-kube-api-access-zqnsg\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.608247 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.608226 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-token\") pod \"e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.676866 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.676793 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:06.812632 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.812606 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q"] Jul 22 09:47:06.813986 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:47:06.813959 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72df0ba_4ba2_4c94_ac0e_124ddc2fb578.slice/crio-ece409b0f8f4d74d683e0deb61c9ef52e09b6c74a603241bd90eb065331e8aef WatchSource:0}: Error finding container ece409b0f8f4d74d683e0deb61c9ef52e09b6c74a603241bd90eb065331e8aef: Status 404 returned error can't find the container with id ece409b0f8f4d74d683e0deb61c9ef52e09b6c74a603241bd90eb065331e8aef Jul 22 09:47:06.815953 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.815923 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:47:06.816020 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.816002 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:47:06.816063 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.816029 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:47:06.828223 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:06.828162 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:07.509233 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:07.509198 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" event={"ID":"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578","Type":"ContainerStarted","Data":"9e06084bed3898207446087b3e41b09c71e4fe9192293f22aaa190a287a2cf3e"} Jul 22 09:47:07.509459 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:07.509253 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" event={"ID":"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578","Type":"ContainerStarted","Data":"ece409b0f8f4d74d683e0deb61c9ef52e09b6c74a603241bd90eb065331e8aef"} Jul 22 09:47:07.530828 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:07.530780 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" podStartSLOduration=1.530765781 podStartE2EDuration="1.530765781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:47:07.529751612 +0000 UTC m=+1757.993546049" watchObservedRunningTime="2026-07-22 09:47:07.530765781 +0000 UTC m=+1757.994560219" Jul 22 09:47:07.677297 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:07.677263 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:07.682166 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:07.682143 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:08.513772 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:08.513744 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:08.514767 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:08.514745 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:08.877293 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:08.877259 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:09.003596 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.003564 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7"] Jul 22 09:47:09.003954 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.003941 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="5de07f5e-bf70-4b82-825f-c7fa6bf6f945" containerName="istio-proxy" Jul 22 09:47:09.006990 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.006975 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7" Jul 22 09:47:09.011819 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.011796 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-d055000b-serving-cert\"" Jul 22 09:47:09.011921 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.011803 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-876r5\"" Jul 22 09:47:09.021495 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.021473 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7"] Jul 22 09:47:09.044136 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.044112 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:09.114528 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.114497 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abddf9e4-2647-417a-8734-b89730d7bc0d-maas-api-tls\") pod \"maas-api-e2e-derive-d055000b-5689cb647c-b4cb7\" (UID: \"abddf9e4-2647-417a-8734-b89730d7bc0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7" Jul 22 09:47:09.114714 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.114553 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abddf9e4-2647-417a-8734-b89730d7bc0d-openshift-service-ca\") pod \"maas-api-e2e-derive-d055000b-5689cb647c-b4cb7\" (UID: \"abddf9e4-2647-417a-8734-b89730d7bc0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7" Jul 22 09:47:09.114714 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.114631 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bqpg\" (UniqueName: \"kubernetes.io/projected/abddf9e4-2647-417a-8734-b89730d7bc0d-kube-api-access-7bqpg\") pod \"maas-api-e2e-derive-d055000b-5689cb647c-b4cb7\" (UID: \"abddf9e4-2647-417a-8734-b89730d7bc0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7" Jul 22 09:47:09.172784 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.172707 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:09.215800 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.215753 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bqpg\" (UniqueName: \"kubernetes.io/projected/abddf9e4-2647-417a-8734-b89730d7bc0d-kube-api-access-7bqpg\") pod \"maas-api-e2e-derive-d055000b-5689cb647c-b4cb7\" (UID: \"abddf9e4-2647-417a-8734-b89730d7bc0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7" Jul 22 09:47:09.216268 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.216241 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abddf9e4-2647-417a-8734-b89730d7bc0d-maas-api-tls\") pod \"maas-api-e2e-derive-d055000b-5689cb647c-b4cb7\" (UID: \"abddf9e4-2647-417a-8734-b89730d7bc0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7" Jul 22 09:47:09.216377 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.216355 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abddf9e4-2647-417a-8734-b89730d7bc0d-openshift-service-ca\") pod \"maas-api-e2e-derive-d055000b-5689cb647c-b4cb7\" (UID: \"abddf9e4-2647-417a-8734-b89730d7bc0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7" Jul 22 09:47:09.217184 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.217157 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abddf9e4-2647-417a-8734-b89730d7bc0d-openshift-service-ca\") pod \"maas-api-e2e-derive-d055000b-5689cb647c-b4cb7\" (UID: \"abddf9e4-2647-417a-8734-b89730d7bc0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7" Jul 22 09:47:09.220175 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.220148 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abddf9e4-2647-417a-8734-b89730d7bc0d-maas-api-tls\") pod \"maas-api-e2e-derive-d055000b-5689cb647c-b4cb7\" (UID: \"abddf9e4-2647-417a-8734-b89730d7bc0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7" Jul 22 09:47:09.236197 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.236168 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bqpg\" (UniqueName: \"kubernetes.io/projected/abddf9e4-2647-417a-8734-b89730d7bc0d-kube-api-access-7bqpg\") pod \"maas-api-e2e-derive-d055000b-5689cb647c-b4cb7\" (UID: \"abddf9e4-2647-417a-8734-b89730d7bc0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7" Jul 22 09:47:09.298832 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.298797 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:09.317004 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.316977 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7" Jul 22 09:47:09.459868 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.459840 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7"] Jul 22 09:47:09.460855 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:47:09.460833 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabddf9e4_2647_417a_8734_b89730d7bc0d.slice/crio-c8d007ca11be0edc1561a5658e54af9408c2a1277dda6fe3209e38745876b622 WatchSource:0}: Error finding container c8d007ca11be0edc1561a5658e54af9408c2a1277dda6fe3209e38745876b622: Status 404 returned error can't find the container with id c8d007ca11be0edc1561a5658e54af9408c2a1277dda6fe3209e38745876b622 Jul 22 09:47:09.519784 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.519743 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7" event={"ID":"abddf9e4-2647-417a-8734-b89730d7bc0d","Type":"ContainerStarted","Data":"c8d007ca11be0edc1561a5658e54af9408c2a1277dda6fe3209e38745876b622"} Jul 22 09:47:09.725183 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.725090 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:09.825966 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:09.825931 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:10.526801 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.526770 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7" event={"ID":"abddf9e4-2647-417a-8734-b89730d7bc0d","Type":"ContainerStarted","Data":"c78ffee0a097f334c906a3fd6a2b3c3420d5505700f1d1a0e5f0c0c0542f2f3d"} Jul 22 09:47:10.544726 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.544677 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7" podStartSLOduration=1.908069157 podStartE2EDuration="2.544662779s" totalImagesPullingTime="636.593622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:47:10.543817724 +0000 UTC m=+1761.007612162" watchObservedRunningTime="2026-07-22 09:47:10.544662779 +0000 UTC m=+1761.008457216" Jul 22 09:47:10.866790 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.866765 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:47:10.929620 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.929589 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a25d30-e365-4d20-8367-469fb10e36a7-istiod-ca-cert\") pod \"25a25d30-e365-4d20-8367-469fb10e36a7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " Jul 22 09:47:10.929808 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.929628 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-workload-socket\") pod \"25a25d30-e365-4d20-8367-469fb10e36a7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " Jul 22 09:47:10.929808 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.929650 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a25d30-e365-4d20-8367-469fb10e36a7-istio-token\") pod \"25a25d30-e365-4d20-8367-469fb10e36a7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " Jul 22 09:47:10.929808 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.929675 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a25d30-e365-4d20-8367-469fb10e36a7-istio-podinfo\") pod \"25a25d30-e365-4d20-8367-469fb10e36a7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " Jul 22 09:47:10.929808 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.929716 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-workload-certs\") pod \"25a25d30-e365-4d20-8367-469fb10e36a7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " Jul 22 09:47:10.929808 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.929784 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-istio-envoy\") pod \"25a25d30-e365-4d20-8367-469fb10e36a7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " Jul 22 09:47:10.930084 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.929816 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-istio-data\") pod \"25a25d30-e365-4d20-8367-469fb10e36a7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " Jul 22 09:47:10.930084 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.929864 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8l4w5\" (UniqueName: \"kubernetes.io/projected/25a25d30-e365-4d20-8367-469fb10e36a7-kube-api-access-8l4w5\") pod \"25a25d30-e365-4d20-8367-469fb10e36a7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " Jul 22 09:47:10.930084 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.929902 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-credential-socket\") pod \"25a25d30-e365-4d20-8367-469fb10e36a7\" (UID: \"25a25d30-e365-4d20-8367-469fb10e36a7\") " Jul 22 09:47:10.930084 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.929910 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "25a25d30-e365-4d20-8367-469fb10e36a7" (UID: "25a25d30-e365-4d20-8367-469fb10e36a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:47:10.930084 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.930045 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "25a25d30-e365-4d20-8367-469fb10e36a7" (UID: "25a25d30-e365-4d20-8367-469fb10e36a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:47:10.930343 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.930120 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25a25d30-e365-4d20-8367-469fb10e36a7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "25a25d30-e365-4d20-8367-469fb10e36a7" (UID: "25a25d30-e365-4d20-8367-469fb10e36a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:47:10.930343 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.930223 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "25a25d30-e365-4d20-8367-469fb10e36a7" (UID: "25a25d30-e365-4d20-8367-469fb10e36a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:47:10.930343 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.930246 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a25d30-e365-4d20-8367-469fb10e36a7-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:10.930343 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.930262 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:10.930343 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.930271 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:10.930813 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.930790 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-istio-data" (OuterVolumeSpecName: "istio-data") pod "25a25d30-e365-4d20-8367-469fb10e36a7" (UID: "25a25d30-e365-4d20-8367-469fb10e36a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:47:10.931951 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.931929 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25a25d30-e365-4d20-8367-469fb10e36a7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "25a25d30-e365-4d20-8367-469fb10e36a7" (UID: "25a25d30-e365-4d20-8367-469fb10e36a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 09:47:10.932033 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.932019 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a25d30-e365-4d20-8367-469fb10e36a7-kube-api-access-8l4w5" (OuterVolumeSpecName: "kube-api-access-8l4w5") pod "25a25d30-e365-4d20-8367-469fb10e36a7" (UID: "25a25d30-e365-4d20-8367-469fb10e36a7"). InnerVolumeSpecName "kube-api-access-8l4w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:47:10.932211 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.932195 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a25d30-e365-4d20-8367-469fb10e36a7-istio-token" (OuterVolumeSpecName: "istio-token") pod "25a25d30-e365-4d20-8367-469fb10e36a7" (UID: "25a25d30-e365-4d20-8367-469fb10e36a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:47:10.932339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:10.932295 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "25a25d30-e365-4d20-8367-469fb10e36a7" (UID: "25a25d30-e365-4d20-8367-469fb10e36a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:47:11.031507 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:11.031477 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:11.031507 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:11.031501 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:11.031507 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:11.031510 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l4w5\" (UniqueName: \"kubernetes.io/projected/25a25d30-e365-4d20-8367-469fb10e36a7-kube-api-access-8l4w5\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:11.031709 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:11.031521 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a25d30-e365-4d20-8367-469fb10e36a7-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:11.031709 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:11.031530 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a25d30-e365-4d20-8367-469fb10e36a7-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:11.031709 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:11.031538 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a25d30-e365-4d20-8367-469fb10e36a7-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:11.533194 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:11.533160 2549 generic.go:356] "Generic (PLEG): container finished" podID="25a25d30-e365-4d20-8367-469fb10e36a7" containerID="9341e27958a874a8fa31f40d1ce40ae6715a1a791ab2e93c00a834d81629d354" exitCode=0 Jul 22 09:47:11.533677 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:11.533205 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" Jul 22 09:47:11.533677 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:11.533248 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" event={"ID":"25a25d30-e365-4d20-8367-469fb10e36a7","Type":"ContainerDied","Data":"9341e27958a874a8fa31f40d1ce40ae6715a1a791ab2e93c00a834d81629d354"} Jul 22 09:47:11.533677 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:11.533331 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7" event={"ID":"25a25d30-e365-4d20-8367-469fb10e36a7","Type":"ContainerDied","Data":"5a48aa698aa6514cfe08cfd5a93ba987a5464a736399af32739f4300afcf79bf"} Jul 22 09:47:11.533677 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:11.533368 2549 scope.go:117] "RemoveContainer" containerID="9341e27958a874a8fa31f40d1ce40ae6715a1a791ab2e93c00a834d81629d354" Jul 22 09:47:11.533677 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:11.533581 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7" Jul 22 09:47:11.542724 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:11.542703 2549 scope.go:117] "RemoveContainer" containerID="0d4005b50a4f89c997939ee633decd7e5eac4b1436db0578ed1f1c4a9950914f" Jul 22 09:47:11.550505 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:11.550487 2549 scope.go:117] "RemoveContainer" containerID="9341e27958a874a8fa31f40d1ce40ae6715a1a791ab2e93c00a834d81629d354" Jul 22 09:47:11.550744 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:47:11.550728 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9341e27958a874a8fa31f40d1ce40ae6715a1a791ab2e93c00a834d81629d354\": container with ID starting with 9341e27958a874a8fa31f40d1ce40ae6715a1a791ab2e93c00a834d81629d354 not found: ID does not exist" containerID="9341e27958a874a8fa31f40d1ce40ae6715a1a791ab2e93c00a834d81629d354" Jul 22 09:47:11.550788 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:11.550770 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9341e27958a874a8fa31f40d1ce40ae6715a1a791ab2e93c00a834d81629d354"} err="failed to get container status \"9341e27958a874a8fa31f40d1ce40ae6715a1a791ab2e93c00a834d81629d354\": rpc error: code = NotFound desc = could not find container \"9341e27958a874a8fa31f40d1ce40ae6715a1a791ab2e93c00a834d81629d354\": container with ID starting with 9341e27958a874a8fa31f40d1ce40ae6715a1a791ab2e93c00a834d81629d354 not found: ID does not exist" Jul 22 09:47:11.550827 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:11.550785 2549 scope.go:117] "RemoveContainer" containerID="0d4005b50a4f89c997939ee633decd7e5eac4b1436db0578ed1f1c4a9950914f" Jul 22 09:47:11.551010 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:47:11.550991 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d4005b50a4f89c997939ee633decd7e5eac4b1436db0578ed1f1c4a9950914f\": container with ID starting with 0d4005b50a4f89c997939ee633decd7e5eac4b1436db0578ed1f1c4a9950914f not found: ID does not exist" containerID="0d4005b50a4f89c997939ee633decd7e5eac4b1436db0578ed1f1c4a9950914f" Jul 22 09:47:11.551070 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:11.551016 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d4005b50a4f89c997939ee633decd7e5eac4b1436db0578ed1f1c4a9950914f"} err="failed to get container status \"0d4005b50a4f89c997939ee633decd7e5eac4b1436db0578ed1f1c4a9950914f\": rpc error: code = NotFound desc = could not find container \"0d4005b50a4f89c997939ee633decd7e5eac4b1436db0578ed1f1c4a9950914f\": container with ID starting with 0d4005b50a4f89c997939ee633decd7e5eac4b1436db0578ed1f1c4a9950914f not found: ID does not exist" Jul 22 09:47:11.577917 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:11.577892 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7"] Jul 22 09:47:11.579755 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:11.579737 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-424990ec-openshift-default-f647c9d95-pxbr7"] Jul 22 09:47:12.340397 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:12.340358 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25a25d30-e365-4d20-8367-469fb10e36a7" path="/var/lib/kubelet/pods/25a25d30-e365-4d20-8367-469fb10e36a7/volumes" Jul 22 09:47:17.544526 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:17.544495 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7" Jul 22 09:47:23.598235 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:23.597629 2549 patch_prober.go:28] interesting pod/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": read tcp 10.133.0.2:54402->10.133.0.43:15021: read: connection reset by peer" start-of-body= Jul 22 09:47:23.598235 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:23.597704 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" podUID="a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": read tcp 10.133.0.2:54402->10.133.0.43:15021: read: connection reset by peer" Jul 22 09:47:24.622470 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:24.622435 2549 generic.go:356] "Generic (PLEG): container finished" podID="a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" containerID="9e06084bed3898207446087b3e41b09c71e4fe9192293f22aaa190a287a2cf3e" exitCode=0 Jul 22 09:47:24.622850 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:24.622516 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" event={"ID":"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578","Type":"ContainerDied","Data":"9e06084bed3898207446087b3e41b09c71e4fe9192293f22aaa190a287a2cf3e"} Jul 22 09:47:24.622922 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:24.622906 2549 scope.go:117] "RemoveContainer" containerID="9e06084bed3898207446087b3e41b09c71e4fe9192293f22aaa190a287a2cf3e" Jul 22 09:47:24.623898 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:24.623863 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:47:24.624006 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:24.623940 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:47:24.624006 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:24.623972 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:47:25.631763 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:25.631725 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" event={"ID":"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578","Type":"ContainerStarted","Data":"d2bd2c5ec26e3701ed1b7f370adf80c208a39343ffe417972d64dc36b15a9cb4"} Jul 22 09:47:25.681697 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:25.681665 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:25.682049 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:25.682026 2549 patch_prober.go:28] interesting pod/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 22 09:47:25.682204 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:25.682180 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" podUID="a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 22 09:47:26.677522 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:26.677477 2549 patch_prober.go:28] interesting pod/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 22 09:47:26.677989 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:26.677548 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" podUID="a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 22 09:47:27.678030 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:27.677990 2549 patch_prober.go:28] interesting pod/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 22 09:47:27.678524 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:27.678053 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" podUID="a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 22 09:47:28.677950 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:28.677890 2549 patch_prober.go:28] interesting pod/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 22 09:47:28.678277 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:28.678244 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" podUID="a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 22 09:47:29.680907 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:29.680876 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:29.681277 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:29.681239 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:29.681835 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:29.681821 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:31.081176 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:31.081139 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:31.173437 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:31.173405 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:31.271466 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:31.271430 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:31.673819 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:31.673788 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:32.048217 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:32.048186 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:32.211635 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:32.211600 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:44.122180 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:44.122101 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7"] Jul 22 09:47:44.122550 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:44.122353 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7" podUID="abddf9e4-2647-417a-8734-b89730d7bc0d" containerName="maas-api" containerID="cri-o://c78ffee0a097f334c906a3fd6a2b3c3420d5505700f1d1a0e5f0c0c0542f2f3d" gracePeriod=30 Jul 22 09:47:44.157297 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:44.157269 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:44.369618 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:44.369592 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7" Jul 22 09:47:44.413998 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:44.413916 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bqpg\" (UniqueName: \"kubernetes.io/projected/abddf9e4-2647-417a-8734-b89730d7bc0d-kube-api-access-7bqpg\") pod \"abddf9e4-2647-417a-8734-b89730d7bc0d\" (UID: \"abddf9e4-2647-417a-8734-b89730d7bc0d\") " Jul 22 09:47:44.413998 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:44.413975 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abddf9e4-2647-417a-8734-b89730d7bc0d-openshift-service-ca\") pod \"abddf9e4-2647-417a-8734-b89730d7bc0d\" (UID: \"abddf9e4-2647-417a-8734-b89730d7bc0d\") " Jul 22 09:47:44.414218 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:44.414031 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abddf9e4-2647-417a-8734-b89730d7bc0d-maas-api-tls\") pod \"abddf9e4-2647-417a-8734-b89730d7bc0d\" (UID: \"abddf9e4-2647-417a-8734-b89730d7bc0d\") " Jul 22 09:47:44.414427 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:44.414402 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abddf9e4-2647-417a-8734-b89730d7bc0d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "abddf9e4-2647-417a-8734-b89730d7bc0d" (UID: "abddf9e4-2647-417a-8734-b89730d7bc0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:47:44.416004 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:44.415977 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abddf9e4-2647-417a-8734-b89730d7bc0d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "abddf9e4-2647-417a-8734-b89730d7bc0d" (UID: "abddf9e4-2647-417a-8734-b89730d7bc0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:47:44.416094 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:44.416079 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abddf9e4-2647-417a-8734-b89730d7bc0d-kube-api-access-7bqpg" (OuterVolumeSpecName: "kube-api-access-7bqpg") pod "abddf9e4-2647-417a-8734-b89730d7bc0d" (UID: "abddf9e4-2647-417a-8734-b89730d7bc0d"). InnerVolumeSpecName "kube-api-access-7bqpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:47:44.515368 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:44.515332 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abddf9e4-2647-417a-8734-b89730d7bc0d-maas-api-tls\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:44.515368 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:44.515363 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bqpg\" (UniqueName: \"kubernetes.io/projected/abddf9e4-2647-417a-8734-b89730d7bc0d-kube-api-access-7bqpg\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:44.515368 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:44.515375 2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abddf9e4-2647-417a-8734-b89730d7bc0d-openshift-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:44.731322 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:44.731207 2549 generic.go:356] "Generic (PLEG): container finished" podID="abddf9e4-2647-417a-8734-b89730d7bc0d" containerID="c78ffee0a097f334c906a3fd6a2b3c3420d5505700f1d1a0e5f0c0c0542f2f3d" exitCode=0 Jul 22 09:47:44.731322 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:44.731251 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7" event={"ID":"abddf9e4-2647-417a-8734-b89730d7bc0d","Type":"ContainerDied","Data":"c78ffee0a097f334c906a3fd6a2b3c3420d5505700f1d1a0e5f0c0c0542f2f3d"} Jul 22 09:47:44.731322 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:44.731279 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7" Jul 22 09:47:44.731596 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:44.731376 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7" event={"ID":"abddf9e4-2647-417a-8734-b89730d7bc0d","Type":"ContainerDied","Data":"c8d007ca11be0edc1561a5658e54af9408c2a1277dda6fe3209e38745876b622"} Jul 22 09:47:44.731596 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:44.731408 2549 scope.go:117] "RemoveContainer" containerID="c78ffee0a097f334c906a3fd6a2b3c3420d5505700f1d1a0e5f0c0c0542f2f3d" Jul 22 09:47:44.740952 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:44.740935 2549 scope.go:117] "RemoveContainer" containerID="c78ffee0a097f334c906a3fd6a2b3c3420d5505700f1d1a0e5f0c0c0542f2f3d" Jul 22 09:47:44.741190 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:47:44.741172 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c78ffee0a097f334c906a3fd6a2b3c3420d5505700f1d1a0e5f0c0c0542f2f3d\": container with ID starting with c78ffee0a097f334c906a3fd6a2b3c3420d5505700f1d1a0e5f0c0c0542f2f3d not found: ID does not exist" containerID="c78ffee0a097f334c906a3fd6a2b3c3420d5505700f1d1a0e5f0c0c0542f2f3d" Jul 22 09:47:44.741240 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:44.741199 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c78ffee0a097f334c906a3fd6a2b3c3420d5505700f1d1a0e5f0c0c0542f2f3d"} err="failed to get container status \"c78ffee0a097f334c906a3fd6a2b3c3420d5505700f1d1a0e5f0c0c0542f2f3d\": rpc error: code = NotFound desc = could not find container \"c78ffee0a097f334c906a3fd6a2b3c3420d5505700f1d1a0e5f0c0c0542f2f3d\": container with ID starting with c78ffee0a097f334c906a3fd6a2b3c3420d5505700f1d1a0e5f0c0c0542f2f3d not found: ID does not exist" Jul 22 09:47:44.752652 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:44.752622 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7"] Jul 22 09:47:44.758783 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:44.758761 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d055000b-5689cb647c-b4cb7"] Jul 22 09:47:46.339649 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:46.339619 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abddf9e4-2647-417a-8734-b89730d7bc0d" path="/var/lib/kubelet/pods/abddf9e4-2647-417a-8734-b89730d7bc0d/volumes" Jul 22 09:47:48.794250 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:48.794216 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:48.935075 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:48.935040 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q"] Jul 22 09:47:48.935342 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:48.935284 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" podUID="a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" containerName="istio-proxy" containerID="cri-o://d2bd2c5ec26e3701ed1b7f370adf80c208a39343ffe417972d64dc36b15a9cb4" gracePeriod=30 Jul 22 09:47:48.974507 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:48.974474 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:49.192563 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:49.192470 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:49.664240 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:49.664207 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:49.810385 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:49.810350 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:50.383947 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:50.383923 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcsgw_dc8d8789-74a0-44ec-8902-44a9650c41ac/ovn-acl-logging/0.log" Jul 22 09:47:50.403115 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:50.403090 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcsgw_dc8d8789-74a0-44ec-8902-44a9650c41ac/ovn-acl-logging/0.log" Jul 22 09:47:51.248450 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:51.248406 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:54.196715 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.196652 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:54.297971 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.297934 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istiod-ca-cert\") pod \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " Jul 22 09:47:54.298129 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.297982 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-credential-socket\") pod \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " Jul 22 09:47:54.298129 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.298027 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-workload-certs\") pod \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " Jul 22 09:47:54.298129 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.298047 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-envoy\") pod \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " Jul 22 09:47:54.298129 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.298077 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-podinfo\") pod \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " Jul 22 09:47:54.298129 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.298103 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-data\") pod \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " Jul 22 09:47:54.298129 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.298120 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqnsg\" (UniqueName: \"kubernetes.io/projected/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-kube-api-access-zqnsg\") pod \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " Jul 22 09:47:54.298426 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.298152 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-token\") pod \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " Jul 22 09:47:54.298426 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.298178 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-workload-socket\") pod \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\" (UID: \"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578\") " Jul 22 09:47:54.298426 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.298382 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" (UID: "a72df0ba-4ba2-4c94-ac0e-124ddc2fb578"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:47:54.298591 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.298472 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" (UID: "a72df0ba-4ba2-4c94-ac0e-124ddc2fb578"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:47:54.298591 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.298504 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" (UID: "a72df0ba-4ba2-4c94-ac0e-124ddc2fb578"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:47:54.298719 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.298695 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:54.298844 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.298726 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:54.298844 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.298746 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:54.298965 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.298869 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" (UID: "a72df0ba-4ba2-4c94-ac0e-124ddc2fb578"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:47:54.299116 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.299092 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-data" (OuterVolumeSpecName: "istio-data") pod "a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" (UID: "a72df0ba-4ba2-4c94-ac0e-124ddc2fb578"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:47:54.300494 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.300470 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-token" (OuterVolumeSpecName: "istio-token") pod "a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" (UID: "a72df0ba-4ba2-4c94-ac0e-124ddc2fb578"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:47:54.300599 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.300518 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-kube-api-access-zqnsg" (OuterVolumeSpecName: "kube-api-access-zqnsg") pod "a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" (UID: "a72df0ba-4ba2-4c94-ac0e-124ddc2fb578"). InnerVolumeSpecName "kube-api-access-zqnsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:47:54.300753 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.300735 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" (UID: "a72df0ba-4ba2-4c94-ac0e-124ddc2fb578"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 09:47:54.300828 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.300809 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" (UID: "a72df0ba-4ba2-4c94-ac0e-124ddc2fb578"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:47:54.399702 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.399670 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:54.399702 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.399695 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:54.399702 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.399703 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:54.399895 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.399712 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqnsg\" (UniqueName: \"kubernetes.io/projected/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-kube-api-access-zqnsg\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:54.399895 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.399722 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:54.399895 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.399731 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:47:54.786769 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.786730 2549 generic.go:356] "Generic (PLEG): container finished" podID="a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" containerID="d2bd2c5ec26e3701ed1b7f370adf80c208a39343ffe417972d64dc36b15a9cb4" exitCode=0 Jul 22 09:47:54.786949 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.786782 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" Jul 22 09:47:54.786949 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.786812 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" event={"ID":"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578","Type":"ContainerDied","Data":"d2bd2c5ec26e3701ed1b7f370adf80c208a39343ffe417972d64dc36b15a9cb4"} Jul 22 09:47:54.786949 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.786861 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q" event={"ID":"a72df0ba-4ba2-4c94-ac0e-124ddc2fb578","Type":"ContainerDied","Data":"ece409b0f8f4d74d683e0deb61c9ef52e09b6c74a603241bd90eb065331e8aef"} Jul 22 09:47:54.786949 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.786884 2549 scope.go:117] "RemoveContainer" containerID="d2bd2c5ec26e3701ed1b7f370adf80c208a39343ffe417972d64dc36b15a9cb4" Jul 22 09:47:54.795625 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.795608 2549 scope.go:117] "RemoveContainer" containerID="9e06084bed3898207446087b3e41b09c71e4fe9192293f22aaa190a287a2cf3e" Jul 22 09:47:54.802993 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.802976 2549 scope.go:117] "RemoveContainer" containerID="d2bd2c5ec26e3701ed1b7f370adf80c208a39343ffe417972d64dc36b15a9cb4" Jul 22 09:47:54.803222 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:47:54.803203 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2bd2c5ec26e3701ed1b7f370adf80c208a39343ffe417972d64dc36b15a9cb4\": container with ID starting with d2bd2c5ec26e3701ed1b7f370adf80c208a39343ffe417972d64dc36b15a9cb4 not found: ID does not exist" containerID="d2bd2c5ec26e3701ed1b7f370adf80c208a39343ffe417972d64dc36b15a9cb4" Jul 22 09:47:54.803274 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.803230 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2bd2c5ec26e3701ed1b7f370adf80c208a39343ffe417972d64dc36b15a9cb4"} err="failed to get container status \"d2bd2c5ec26e3701ed1b7f370adf80c208a39343ffe417972d64dc36b15a9cb4\": rpc error: code = NotFound desc = could not find container \"d2bd2c5ec26e3701ed1b7f370adf80c208a39343ffe417972d64dc36b15a9cb4\": container with ID starting with d2bd2c5ec26e3701ed1b7f370adf80c208a39343ffe417972d64dc36b15a9cb4 not found: ID does not exist" Jul 22 09:47:54.803274 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.803246 2549 scope.go:117] "RemoveContainer" containerID="9e06084bed3898207446087b3e41b09c71e4fe9192293f22aaa190a287a2cf3e" Jul 22 09:47:54.803520 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:47:54.803501 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e06084bed3898207446087b3e41b09c71e4fe9192293f22aaa190a287a2cf3e\": container with ID starting with 9e06084bed3898207446087b3e41b09c71e4fe9192293f22aaa190a287a2cf3e not found: ID does not exist" containerID="9e06084bed3898207446087b3e41b09c71e4fe9192293f22aaa190a287a2cf3e" Jul 22 09:47:54.803589 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.803526 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e06084bed3898207446087b3e41b09c71e4fe9192293f22aaa190a287a2cf3e"} err="failed to get container status \"9e06084bed3898207446087b3e41b09c71e4fe9192293f22aaa190a287a2cf3e\": rpc error: code = NotFound desc = could not find container \"9e06084bed3898207446087b3e41b09c71e4fe9192293f22aaa190a287a2cf3e\": container with ID starting with 9e06084bed3898207446087b3e41b09c71e4fe9192293f22aaa190a287a2cf3e not found: ID does not exist" Jul 22 09:47:54.808591 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.808571 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q"] Jul 22 09:47:54.814906 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:54.814887 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-d055000b-openshift-default-7c7548b79f-hpz7q"] Jul 22 09:47:56.338959 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:56.338927 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" path="/var/lib/kubelet/pods/a72df0ba-4ba2-4c94-ac0e-124ddc2fb578/volumes" Jul 22 09:47:57.128674 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:57.128635 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:57.240224 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:57.240194 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:57.364871 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:57.364836 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:57.466564 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:57.466485 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:47:57.821969 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:47:57.821938 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:48:23.739731 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:48:23.739695 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:48:30.188384 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:48:30.188344 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:48:30.290460 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:48:30.290426 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:48:30.389736 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:48:30.389705 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:48:46.288248 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:48:46.288204 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:48:52.611213 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:48:52.611171 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:48:52.710858 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:48:52.710823 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:03.238732 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:03.238698 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:04.087396 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:04.087355 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:11.061754 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:11.061719 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:11.176469 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:11.176438 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:11.286863 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:11.286831 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:11.386521 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:11.386432 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:12.166236 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:12.166207 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:12.270357 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:12.270320 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:12.365970 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:12.365932 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:13.975521 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:13.975490 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:20.091296 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:20.091262 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:20.190943 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:20.190907 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:20.732222 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:20.732183 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:27.143681 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.143646 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:27.248820 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.248784 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:27.374951 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.374917 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:27.694658 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.694628 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59"] Jul 22 09:49:27.694991 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.694975 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abddf9e4-2647-417a-8734-b89730d7bc0d" containerName="maas-api" Jul 22 09:49:27.695079 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.694994 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="abddf9e4-2647-417a-8734-b89730d7bc0d" containerName="maas-api" Jul 22 09:49:27.695079 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.695008 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25a25d30-e365-4d20-8367-469fb10e36a7" containerName="istio-proxy" Jul 22 09:49:27.695079 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.695015 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="25a25d30-e365-4d20-8367-469fb10e36a7" containerName="istio-proxy" Jul 22 09:49:27.695079 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.695031 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" containerName="istio-proxy" Jul 22 09:49:27.695079 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.695040 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" containerName="istio-proxy" Jul 22 09:49:27.695079 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.695055 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25a25d30-e365-4d20-8367-469fb10e36a7" containerName="istio-proxy" Jul 22 09:49:27.695079 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.695062 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="25a25d30-e365-4d20-8367-469fb10e36a7" containerName="istio-proxy" Jul 22 09:49:27.695444 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.695083 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" containerName="istio-proxy" Jul 22 09:49:27.695444 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.695093 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" containerName="istio-proxy" Jul 22 09:49:27.695444 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.695169 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="25a25d30-e365-4d20-8367-469fb10e36a7" containerName="istio-proxy" Jul 22 09:49:27.695444 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.695180 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="abddf9e4-2647-417a-8734-b89730d7bc0d" containerName="maas-api" Jul 22 09:49:27.695444 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.695192 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" containerName="istio-proxy" Jul 22 09:49:27.698273 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.698251 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.701149 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.701124 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5ba85bad-openshift-default-dockercfg-p6nqs\"" Jul 22 09:49:27.728015 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.727978 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59"] Jul 22 09:49:27.761567 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.761524 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:27.785133 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.785105 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-envoy\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.785252 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.785135 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-token\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.785252 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.785157 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b315b4a8-a393-4c90-b568-bf6de7d71547-istiod-ca-cert\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.785252 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.785172 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-workload-socket\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.785387 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.785252 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-workload-certs\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.785387 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.785278 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-data\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.785387 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.785331 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-podinfo\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.785387 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.785376 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-credential-socket\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.785511 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.785410 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p22dd\" (UniqueName: \"kubernetes.io/projected/b315b4a8-a393-4c90-b568-bf6de7d71547-kube-api-access-p22dd\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.835272 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.835237 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:27.886174 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.886142 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-podinfo\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.886415 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.886191 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-credential-socket\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.886415 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.886219 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p22dd\" (UniqueName: \"kubernetes.io/projected/b315b4a8-a393-4c90-b568-bf6de7d71547-kube-api-access-p22dd\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.886415 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.886296 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-envoy\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.886415 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.886335 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-token\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.886636 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.886493 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b315b4a8-a393-4c90-b568-bf6de7d71547-istiod-ca-cert\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.886636 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.886537 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-workload-socket\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.886636 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.886588 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-workload-certs\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.886636 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.886609 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-credential-socket\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.886636 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.886613 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-data\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.886905 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.886882 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-workload-socket\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.886969 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.886935 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-data\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.887033 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.887011 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-workload-certs\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.887177 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.887159 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b315b4a8-a393-4c90-b568-bf6de7d71547-istiod-ca-cert\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.888552 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.888522 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-envoy\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.888757 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.888739 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-podinfo\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.894885 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.894861 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-token\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:27.895204 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:27.895181 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p22dd\" (UniqueName: \"kubernetes.io/projected/b315b4a8-a393-4c90-b568-bf6de7d71547-kube-api-access-p22dd\") pod \"e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:28.008146 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:28.008049 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:28.146631 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:28.146605 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59"] Jul 22 09:49:28.148489 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:49:28.148464 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb315b4a8_a393_4c90_b568_bf6de7d71547.slice/crio-0a1ae4b5271a83e27a5c85734327be84df43f82af9fa2608fef6ea7a4deafe9d WatchSource:0}: Error finding container 0a1ae4b5271a83e27a5c85734327be84df43f82af9fa2608fef6ea7a4deafe9d: Status 404 returned error can't find the container with id 0a1ae4b5271a83e27a5c85734327be84df43f82af9fa2608fef6ea7a4deafe9d Jul 22 09:49:28.150620 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:28.150590 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:49:28.150676 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:28.150651 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:49:28.150716 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:28.150680 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:49:28.240361 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:28.240321 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" event={"ID":"b315b4a8-a393-4c90-b568-bf6de7d71547","Type":"ContainerStarted","Data":"4fe79235b34b893c1f7ff024119a36991ab3c7b6d83c115091fdcd458316395b"} Jul 22 09:49:28.240557 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:28.240393 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" event={"ID":"b315b4a8-a393-4c90-b568-bf6de7d71547","Type":"ContainerStarted","Data":"0a1ae4b5271a83e27a5c85734327be84df43f82af9fa2608fef6ea7a4deafe9d"} Jul 22 09:49:28.267180 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:28.267082 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" podStartSLOduration=1.2670614709999999 podStartE2EDuration="1.267061471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:49:28.264185796 +0000 UTC m=+1898.727980236" watchObservedRunningTime="2026-07-22 09:49:28.267061471 +0000 UTC m=+1898.730855912" Jul 22 09:49:29.009208 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:29.009176 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:29.014011 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:29.013989 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:29.245709 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:29.245673 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:29.246597 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:29.246580 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:30.535419 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:30.535379 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:30.639590 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:30.639551 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:30.648339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:30.648298 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg"] Jul 22 09:49:30.648694 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:30.648681 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="a72df0ba-4ba2-4c94-ac0e-124ddc2fb578" containerName="istio-proxy" Jul 22 09:49:30.648760 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:30.648697 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="25a25d30-e365-4d20-8367-469fb10e36a7" containerName="istio-proxy" Jul 22 09:49:30.651629 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:30.651611 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg" Jul 22 09:49:30.654129 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:30.654110 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5ba85bad-serving-cert\"" Jul 22 09:49:30.654277 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:30.654261 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-876r5\"" Jul 22 09:49:30.662537 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:30.662513 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg"] Jul 22 09:49:30.708128 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:30.708092 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx4ps\" (UniqueName: \"kubernetes.io/projected/916e31df-0846-425e-be6e-3461cf0ed5b8-kube-api-access-nx4ps\") pod \"maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg\" (UID: \"916e31df-0846-425e-be6e-3461cf0ed5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg" Jul 22 09:49:30.708128 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:30.708125 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/916e31df-0846-425e-be6e-3461cf0ed5b8-maas-api-tls\") pod \"maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg\" (UID: \"916e31df-0846-425e-be6e-3461cf0ed5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg" Jul 22 09:49:30.708357 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:30.708167 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/916e31df-0846-425e-be6e-3461cf0ed5b8-openshift-service-ca\") pod \"maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg\" (UID: \"916e31df-0846-425e-be6e-3461cf0ed5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg" Jul 22 09:49:30.736891 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:30.736860 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:30.808737 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:30.808602 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx4ps\" (UniqueName: \"kubernetes.io/projected/916e31df-0846-425e-be6e-3461cf0ed5b8-kube-api-access-nx4ps\") pod \"maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg\" (UID: \"916e31df-0846-425e-be6e-3461cf0ed5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg" Jul 22 09:49:30.808737 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:30.808642 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/916e31df-0846-425e-be6e-3461cf0ed5b8-maas-api-tls\") pod \"maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg\" (UID: \"916e31df-0846-425e-be6e-3461cf0ed5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg" Jul 22 09:49:30.808737 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:30.808694 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/916e31df-0846-425e-be6e-3461cf0ed5b8-openshift-service-ca\") pod \"maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg\" (UID: \"916e31df-0846-425e-be6e-3461cf0ed5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg" Jul 22 09:49:30.809472 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:30.809449 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/916e31df-0846-425e-be6e-3461cf0ed5b8-openshift-service-ca\") pod \"maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg\" (UID: \"916e31df-0846-425e-be6e-3461cf0ed5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg" Jul 22 09:49:30.811260 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:30.811237 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/916e31df-0846-425e-be6e-3461cf0ed5b8-maas-api-tls\") pod \"maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg\" (UID: \"916e31df-0846-425e-be6e-3461cf0ed5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg" Jul 22 09:49:30.820457 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:30.820434 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx4ps\" (UniqueName: \"kubernetes.io/projected/916e31df-0846-425e-be6e-3461cf0ed5b8-kube-api-access-nx4ps\") pod \"maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg\" (UID: \"916e31df-0846-425e-be6e-3461cf0ed5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg" Jul 22 09:49:30.848500 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:30.848476 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:30.963453 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:30.963421 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg" Jul 22 09:49:31.096969 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:31.096944 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg"] Jul 22 09:49:31.098540 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:49:31.098517 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916e31df_0846_425e_be6e_3461cf0ed5b8.slice/crio-bf2a525e7b31ac94582082f00a306908be4b0258b5a1bb1cc2fffa2567e035bf WatchSource:0}: Error finding container bf2a525e7b31ac94582082f00a306908be4b0258b5a1bb1cc2fffa2567e035bf: Status 404 returned error can't find the container with id bf2a525e7b31ac94582082f00a306908be4b0258b5a1bb1cc2fffa2567e035bf Jul 22 09:49:31.256614 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:31.256576 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:31.257645 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:31.257616 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg" event={"ID":"916e31df-0846-425e-be6e-3461cf0ed5b8","Type":"ContainerStarted","Data":"bf2a525e7b31ac94582082f00a306908be4b0258b5a1bb1cc2fffa2567e035bf"} Jul 22 09:49:31.746905 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:31.746868 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:32.263956 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:32.263918 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg" event={"ID":"916e31df-0846-425e-be6e-3461cf0ed5b8","Type":"ContainerStarted","Data":"7cd4252b157705db07246575d07ebc74332b0362414308ce0542e0a6c5ddee92"} Jul 22 09:49:32.264146 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:32.264041 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg" Jul 22 09:49:32.284278 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:32.284229 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg" podStartSLOduration=1.856995129 podStartE2EDuration="2.284214661s" totalImagesPullingTime="427.219532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:49:32.282739498 +0000 UTC m=+1902.746533936" watchObservedRunningTime="2026-07-22 09:49:32.284214661 +0000 UTC m=+1902.748009098" Jul 22 09:49:34.932186 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:34.932145 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:35.549023 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:35.548975 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:35.645560 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:35.645526 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:36.006378 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:36.006272 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:37.797834 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:37.797801 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:38.284698 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:38.284658 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg" Jul 22 09:49:42.019873 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:42.019783 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:42.096081 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:42.096045 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:42.324568 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:42.324533 2549 generic.go:356] "Generic (PLEG): container finished" podID="b315b4a8-a393-4c90-b568-bf6de7d71547" containerID="4fe79235b34b893c1f7ff024119a36991ab3c7b6d83c115091fdcd458316395b" exitCode=0 Jul 22 09:49:42.324731 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:42.324599 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" event={"ID":"b315b4a8-a393-4c90-b568-bf6de7d71547","Type":"ContainerDied","Data":"4fe79235b34b893c1f7ff024119a36991ab3c7b6d83c115091fdcd458316395b"} Jul 22 09:49:42.324946 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:42.324934 2549 scope.go:117] "RemoveContainer" containerID="4fe79235b34b893c1f7ff024119a36991ab3c7b6d83c115091fdcd458316395b" Jul 22 09:49:42.325885 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:42.325855 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:49:42.325979 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:42.325938 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:49:42.326040 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:42.325988 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:49:43.330840 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:43.330807 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" event={"ID":"b315b4a8-a393-4c90-b568-bf6de7d71547","Type":"ContainerStarted","Data":"8050bd09e7c91a62dc151c0c8e7555eedb9bfab7c76c50a15e0985f37fda9311"} Jul 22 09:49:44.009102 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:44.009065 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:44.010248 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:44.010221 2549 patch_prober.go:28] interesting pod/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 22 09:49:44.010388 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:44.010285 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" podUID="b315b4a8-a393-4c90-b568-bf6de7d71547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 22 09:49:44.246388 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:44.246351 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:45.008878 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:45.008844 2549 patch_prober.go:28] interesting pod/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 22 09:49:45.009261 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:45.008898 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" podUID="b315b4a8-a393-4c90-b568-bf6de7d71547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 22 09:49:46.008579 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:46.008541 2549 patch_prober.go:28] interesting pod/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 22 09:49:46.008766 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:46.008610 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" podUID="b315b4a8-a393-4c90-b568-bf6de7d71547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 22 09:49:47.009269 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:47.009229 2549 patch_prober.go:28] interesting pod/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 22 09:49:47.009690 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:47.009325 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" podUID="b315b4a8-a393-4c90-b568-bf6de7d71547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 22 09:49:48.012375 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:48.012335 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:48.013247 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:48.013229 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:49:52.830565 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:52.830531 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg"] Jul 22 09:49:52.830987 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:52.830736 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg" podUID="916e31df-0846-425e-be6e-3461cf0ed5b8" containerName="maas-api" containerID="cri-o://7cd4252b157705db07246575d07ebc74332b0362414308ce0542e0a6c5ddee92" gracePeriod=30 Jul 22 09:49:52.844471 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:52.844440 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:53.074451 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:53.074428 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg" Jul 22 09:49:53.215632 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:53.215548 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/916e31df-0846-425e-be6e-3461cf0ed5b8-maas-api-tls\") pod \"916e31df-0846-425e-be6e-3461cf0ed5b8\" (UID: \"916e31df-0846-425e-be6e-3461cf0ed5b8\") " Jul 22 09:49:53.215632 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:53.215585 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nx4ps\" (UniqueName: \"kubernetes.io/projected/916e31df-0846-425e-be6e-3461cf0ed5b8-kube-api-access-nx4ps\") pod \"916e31df-0846-425e-be6e-3461cf0ed5b8\" (UID: \"916e31df-0846-425e-be6e-3461cf0ed5b8\") " Jul 22 09:49:53.215852 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:53.215666 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/916e31df-0846-425e-be6e-3461cf0ed5b8-openshift-service-ca\") pod \"916e31df-0846-425e-be6e-3461cf0ed5b8\" (UID: \"916e31df-0846-425e-be6e-3461cf0ed5b8\") " Jul 22 09:49:53.216078 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:53.216042 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916e31df-0846-425e-be6e-3461cf0ed5b8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "916e31df-0846-425e-be6e-3461cf0ed5b8" (UID: "916e31df-0846-425e-be6e-3461cf0ed5b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:49:53.217667 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:53.217644 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916e31df-0846-425e-be6e-3461cf0ed5b8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "916e31df-0846-425e-be6e-3461cf0ed5b8" (UID: "916e31df-0846-425e-be6e-3461cf0ed5b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:49:53.217758 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:53.217701 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916e31df-0846-425e-be6e-3461cf0ed5b8-kube-api-access-nx4ps" (OuterVolumeSpecName: "kube-api-access-nx4ps") pod "916e31df-0846-425e-be6e-3461cf0ed5b8" (UID: "916e31df-0846-425e-be6e-3461cf0ed5b8"). InnerVolumeSpecName "kube-api-access-nx4ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:49:53.316555 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:53.316520 2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/916e31df-0846-425e-be6e-3461cf0ed5b8-openshift-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:49:53.316555 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:53.316552 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/916e31df-0846-425e-be6e-3461cf0ed5b8-maas-api-tls\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:49:53.316737 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:53.316565 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx4ps\" (UniqueName: \"kubernetes.io/projected/916e31df-0846-425e-be6e-3461cf0ed5b8-kube-api-access-nx4ps\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:49:53.381628 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:53.381593 2549 generic.go:356] "Generic (PLEG): container finished" podID="916e31df-0846-425e-be6e-3461cf0ed5b8" containerID="7cd4252b157705db07246575d07ebc74332b0362414308ce0542e0a6c5ddee92" exitCode=0 Jul 22 09:49:53.381788 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:53.381642 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg" Jul 22 09:49:53.381788 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:53.381689 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg" event={"ID":"916e31df-0846-425e-be6e-3461cf0ed5b8","Type":"ContainerDied","Data":"7cd4252b157705db07246575d07ebc74332b0362414308ce0542e0a6c5ddee92"} Jul 22 09:49:53.381788 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:53.381744 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg" event={"ID":"916e31df-0846-425e-be6e-3461cf0ed5b8","Type":"ContainerDied","Data":"bf2a525e7b31ac94582082f00a306908be4b0258b5a1bb1cc2fffa2567e035bf"} Jul 22 09:49:53.381788 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:53.381770 2549 scope.go:117] "RemoveContainer" containerID="7cd4252b157705db07246575d07ebc74332b0362414308ce0542e0a6c5ddee92" Jul 22 09:49:53.390723 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:53.390704 2549 scope.go:117] "RemoveContainer" containerID="7cd4252b157705db07246575d07ebc74332b0362414308ce0542e0a6c5ddee92" Jul 22 09:49:53.390968 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:49:53.390949 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd4252b157705db07246575d07ebc74332b0362414308ce0542e0a6c5ddee92\": container with ID starting with 7cd4252b157705db07246575d07ebc74332b0362414308ce0542e0a6c5ddee92 not found: ID does not exist" containerID="7cd4252b157705db07246575d07ebc74332b0362414308ce0542e0a6c5ddee92" Jul 22 09:49:53.391025 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:53.390974 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd4252b157705db07246575d07ebc74332b0362414308ce0542e0a6c5ddee92"} err="failed to get container status \"7cd4252b157705db07246575d07ebc74332b0362414308ce0542e0a6c5ddee92\": rpc error: code = NotFound desc = could not find container \"7cd4252b157705db07246575d07ebc74332b0362414308ce0542e0a6c5ddee92\": container with ID starting with 7cd4252b157705db07246575d07ebc74332b0362414308ce0542e0a6c5ddee92 not found: ID does not exist" Jul 22 09:49:53.402264 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:53.402243 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg"] Jul 22 09:49:53.406077 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:53.406058 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ba85bad-86764c4f7f-bx5sg"] Jul 22 09:49:54.339412 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:54.339381 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="916e31df-0846-425e-be6e-3461cf0ed5b8" path="/var/lib/kubelet/pods/916e31df-0846-425e-be6e-3461cf0ed5b8/volumes" Jul 22 09:49:57.454682 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:57.454646 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:49:58.636920 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:49:58.636885 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:04.182447 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:04.182410 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:04.271502 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:04.271464 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:04.423222 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:04.423186 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59"] Jul 22 09:50:04.423437 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:04.423415 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" podUID="b315b4a8-a393-4c90-b568-bf6de7d71547" containerName="istio-proxy" containerID="cri-o://8050bd09e7c91a62dc151c0c8e7555eedb9bfab7c76c50a15e0985f37fda9311" gracePeriod=30 Jul 22 09:50:04.435041 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:04.434976 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:04.657761 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:04.657716 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:04.807944 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:04.807900 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:04.912626 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:04.912592 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:05.110382 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:05.110271 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:09.690871 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.690811 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:50:09.860606 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.860569 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b315b4a8-a393-4c90-b568-bf6de7d71547-istiod-ca-cert\") pod \"b315b4a8-a393-4c90-b568-bf6de7d71547\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " Jul 22 09:50:09.860771 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.860616 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-token\") pod \"b315b4a8-a393-4c90-b568-bf6de7d71547\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " Jul 22 09:50:09.860771 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.860635 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-workload-certs\") pod \"b315b4a8-a393-4c90-b568-bf6de7d71547\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " Jul 22 09:50:09.860771 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.860684 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-podinfo\") pod \"b315b4a8-a393-4c90-b568-bf6de7d71547\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " Jul 22 09:50:09.860771 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.860704 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-credential-socket\") pod \"b315b4a8-a393-4c90-b568-bf6de7d71547\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " Jul 22 09:50:09.860771 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.860732 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-envoy\") pod \"b315b4a8-a393-4c90-b568-bf6de7d71547\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " Jul 22 09:50:09.860771 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.860772 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p22dd\" (UniqueName: \"kubernetes.io/projected/b315b4a8-a393-4c90-b568-bf6de7d71547-kube-api-access-p22dd\") pod \"b315b4a8-a393-4c90-b568-bf6de7d71547\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " Jul 22 09:50:09.861013 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.860800 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-data\") pod \"b315b4a8-a393-4c90-b568-bf6de7d71547\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " Jul 22 09:50:09.861013 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.860825 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-workload-socket\") pod \"b315b4a8-a393-4c90-b568-bf6de7d71547\" (UID: \"b315b4a8-a393-4c90-b568-bf6de7d71547\") " Jul 22 09:50:09.861013 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.860980 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b315b4a8-a393-4c90-b568-bf6de7d71547-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b315b4a8-a393-4c90-b568-bf6de7d71547" (UID: "b315b4a8-a393-4c90-b568-bf6de7d71547"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:50:09.861128 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.861077 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b315b4a8-a393-4c90-b568-bf6de7d71547-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:50:09.861379 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.861295 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b315b4a8-a393-4c90-b568-bf6de7d71547" (UID: "b315b4a8-a393-4c90-b568-bf6de7d71547"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:50:09.861510 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.861403 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b315b4a8-a393-4c90-b568-bf6de7d71547" (UID: "b315b4a8-a393-4c90-b568-bf6de7d71547"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:50:09.861510 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.861394 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b315b4a8-a393-4c90-b568-bf6de7d71547" (UID: "b315b4a8-a393-4c90-b568-bf6de7d71547"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:50:09.861882 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.861850 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-data" (OuterVolumeSpecName: "istio-data") pod "b315b4a8-a393-4c90-b568-bf6de7d71547" (UID: "b315b4a8-a393-4c90-b568-bf6de7d71547"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:50:09.862895 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.862869 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-token" (OuterVolumeSpecName: "istio-token") pod "b315b4a8-a393-4c90-b568-bf6de7d71547" (UID: "b315b4a8-a393-4c90-b568-bf6de7d71547"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:50:09.863138 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.863119 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b315b4a8-a393-4c90-b568-bf6de7d71547" (UID: "b315b4a8-a393-4c90-b568-bf6de7d71547"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:50:09.863317 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.863272 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b315b4a8-a393-4c90-b568-bf6de7d71547-kube-api-access-p22dd" (OuterVolumeSpecName: "kube-api-access-p22dd") pod "b315b4a8-a393-4c90-b568-bf6de7d71547" (UID: "b315b4a8-a393-4c90-b568-bf6de7d71547"). InnerVolumeSpecName "kube-api-access-p22dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:50:09.863317 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.863285 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b315b4a8-a393-4c90-b568-bf6de7d71547" (UID: "b315b4a8-a393-4c90-b568-bf6de7d71547"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 09:50:09.962153 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.962067 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:50:09.962153 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.962098 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:50:09.962153 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.962109 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:50:09.962153 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.962119 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p22dd\" (UniqueName: \"kubernetes.io/projected/b315b4a8-a393-4c90-b568-bf6de7d71547-kube-api-access-p22dd\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:50:09.962153 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.962129 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:50:09.962153 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.962139 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:50:09.962153 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.962147 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b315b4a8-a393-4c90-b568-bf6de7d71547-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:50:09.962153 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:09.962155 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b315b4a8-a393-4c90-b568-bf6de7d71547-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:50:10.466581 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:10.466548 2549 generic.go:356] "Generic (PLEG): container finished" podID="b315b4a8-a393-4c90-b568-bf6de7d71547" containerID="8050bd09e7c91a62dc151c0c8e7555eedb9bfab7c76c50a15e0985f37fda9311" exitCode=0 Jul 22 09:50:10.466763 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:10.466587 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" event={"ID":"b315b4a8-a393-4c90-b568-bf6de7d71547","Type":"ContainerDied","Data":"8050bd09e7c91a62dc151c0c8e7555eedb9bfab7c76c50a15e0985f37fda9311"} Jul 22 09:50:10.466763 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:10.466610 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" Jul 22 09:50:10.466763 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:10.466643 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59" event={"ID":"b315b4a8-a393-4c90-b568-bf6de7d71547","Type":"ContainerDied","Data":"0a1ae4b5271a83e27a5c85734327be84df43f82af9fa2608fef6ea7a4deafe9d"} Jul 22 09:50:10.466763 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:10.466666 2549 scope.go:117] "RemoveContainer" containerID="8050bd09e7c91a62dc151c0c8e7555eedb9bfab7c76c50a15e0985f37fda9311" Jul 22 09:50:10.475505 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:10.475490 2549 scope.go:117] "RemoveContainer" containerID="4fe79235b34b893c1f7ff024119a36991ab3c7b6d83c115091fdcd458316395b" Jul 22 09:50:10.483039 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:10.483019 2549 scope.go:117] "RemoveContainer" containerID="8050bd09e7c91a62dc151c0c8e7555eedb9bfab7c76c50a15e0985f37fda9311" Jul 22 09:50:10.483324 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:50:10.483289 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8050bd09e7c91a62dc151c0c8e7555eedb9bfab7c76c50a15e0985f37fda9311\": container with ID starting with 8050bd09e7c91a62dc151c0c8e7555eedb9bfab7c76c50a15e0985f37fda9311 not found: ID does not exist" containerID="8050bd09e7c91a62dc151c0c8e7555eedb9bfab7c76c50a15e0985f37fda9311" Jul 22 09:50:10.483426 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:10.483346 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8050bd09e7c91a62dc151c0c8e7555eedb9bfab7c76c50a15e0985f37fda9311"} err="failed to get container status \"8050bd09e7c91a62dc151c0c8e7555eedb9bfab7c76c50a15e0985f37fda9311\": rpc error: code = NotFound desc = could not find container \"8050bd09e7c91a62dc151c0c8e7555eedb9bfab7c76c50a15e0985f37fda9311\": container with ID starting with 8050bd09e7c91a62dc151c0c8e7555eedb9bfab7c76c50a15e0985f37fda9311 not found: ID does not exist" Jul 22 09:50:10.483426 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:10.483370 2549 scope.go:117] "RemoveContainer" containerID="4fe79235b34b893c1f7ff024119a36991ab3c7b6d83c115091fdcd458316395b" Jul 22 09:50:10.483679 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:50:10.483655 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fe79235b34b893c1f7ff024119a36991ab3c7b6d83c115091fdcd458316395b\": container with ID starting with 4fe79235b34b893c1f7ff024119a36991ab3c7b6d83c115091fdcd458316395b not found: ID does not exist" containerID="4fe79235b34b893c1f7ff024119a36991ab3c7b6d83c115091fdcd458316395b" Jul 22 09:50:10.483730 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:10.483689 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fe79235b34b893c1f7ff024119a36991ab3c7b6d83c115091fdcd458316395b"} err="failed to get container status \"4fe79235b34b893c1f7ff024119a36991ab3c7b6d83c115091fdcd458316395b\": rpc error: code = NotFound desc = could not find container \"4fe79235b34b893c1f7ff024119a36991ab3c7b6d83c115091fdcd458316395b\": container with ID starting with 4fe79235b34b893c1f7ff024119a36991ab3c7b6d83c115091fdcd458316395b not found: ID does not exist" Jul 22 09:50:10.489327 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:10.489263 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59"] Jul 22 09:50:10.491990 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:10.491971 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5ba85bad-openshift-default-d9dfb544c-2xw59"] Jul 22 09:50:10.933634 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:10.933606 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:11.912838 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:11.912806 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:12.016507 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:12.016474 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:12.340112 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:12.340083 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b315b4a8-a393-4c90-b568-bf6de7d71547" path="/var/lib/kubelet/pods/b315b4a8-a393-4c90-b568-bf6de7d71547/volumes" Jul 22 09:50:17.369600 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:17.369560 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:17.428480 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:17.428426 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:24.093946 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.093908 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:24.385761 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.385686 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st"] Jul 22 09:50:24.385984 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.385971 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b315b4a8-a393-4c90-b568-bf6de7d71547" containerName="istio-proxy" Jul 22 09:50:24.386026 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.385986 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b315b4a8-a393-4c90-b568-bf6de7d71547" containerName="istio-proxy" Jul 22 09:50:24.386026 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.385995 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="916e31df-0846-425e-be6e-3461cf0ed5b8" containerName="maas-api" Jul 22 09:50:24.386026 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.386001 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="916e31df-0846-425e-be6e-3461cf0ed5b8" containerName="maas-api" Jul 22 09:50:24.386026 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.386016 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b315b4a8-a393-4c90-b568-bf6de7d71547" containerName="istio-proxy" Jul 22 09:50:24.386026 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.386022 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b315b4a8-a393-4c90-b568-bf6de7d71547" containerName="istio-proxy" Jul 22 09:50:24.386179 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.386070 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="b315b4a8-a393-4c90-b568-bf6de7d71547" containerName="istio-proxy" Jul 22 09:50:24.386179 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.386078 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="b315b4a8-a393-4c90-b568-bf6de7d71547" containerName="istio-proxy" Jul 22 09:50:24.386179 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.386085 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="916e31df-0846-425e-be6e-3461cf0ed5b8" containerName="maas-api" Jul 22 09:50:24.388894 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.388878 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.391896 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.391875 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6285b203-openshift-default-dockercfg-rhrj4\"" Jul 22 09:50:24.405663 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.405641 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st"] Jul 22 09:50:24.416479 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.416457 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:24.478702 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.478666 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-data\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.478861 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.478712 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-podinfo\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.478861 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.478741 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-credential-socket\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.478861 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.478782 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-token\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.478861 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.478836 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eacbe5e-a91a-4674-a5ee-f0619022826d-istiod-ca-cert\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.479010 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.478899 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-workload-certs\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.479010 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.478921 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-envoy\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.479010 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.478956 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck8xh\" (UniqueName: \"kubernetes.io/projected/3eacbe5e-a91a-4674-a5ee-f0619022826d-kube-api-access-ck8xh\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.479123 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.479039 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-workload-socket\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.567735 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.567699 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:24.580042 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.580016 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-workload-certs\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.580148 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.580050 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-envoy\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.580148 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.580076 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck8xh\" (UniqueName: \"kubernetes.io/projected/3eacbe5e-a91a-4674-a5ee-f0619022826d-kube-api-access-ck8xh\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.580148 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.580114 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-workload-socket\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.580148 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.580133 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-data\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.580400 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.580154 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-podinfo\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.580400 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.580180 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-credential-socket\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.580400 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.580200 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-token\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.580400 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.580231 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eacbe5e-a91a-4674-a5ee-f0619022826d-istiod-ca-cert\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.580613 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.580518 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-workload-certs\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.580613 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.580555 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-workload-socket\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.580721 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.580676 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-data\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.580887 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.580867 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-credential-socket\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.580963 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.580911 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eacbe5e-a91a-4674-a5ee-f0619022826d-istiod-ca-cert\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.582342 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.582323 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-envoy\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.582578 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.582560 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-podinfo\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.588288 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.588259 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-token\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.588435 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.588417 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck8xh\" (UniqueName: \"kubernetes.io/projected/3eacbe5e-a91a-4674-a5ee-f0619022826d-kube-api-access-ck8xh\") pod \"e2e-mt-6285b203-openshift-default-df855676c-mz8st\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.698677 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.698590 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:24.838448 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.838415 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st"] Jul 22 09:50:24.840258 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:50:24.840227 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eacbe5e_a91a_4674_a5ee_f0619022826d.slice/crio-ca3f837899fbb59382046bdca7c463dd18a30166ade0254b46419f7bae4c48a8 WatchSource:0}: Error finding container ca3f837899fbb59382046bdca7c463dd18a30166ade0254b46419f7bae4c48a8: Status 404 returned error can't find the container with id ca3f837899fbb59382046bdca7c463dd18a30166ade0254b46419f7bae4c48a8 Jul 22 09:50:24.842605 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.842569 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:50:24.842712 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.842657 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:50:24.842712 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:24.842690 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:50:25.539635 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:25.539601 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" event={"ID":"3eacbe5e-a91a-4674-a5ee-f0619022826d","Type":"ContainerStarted","Data":"1a796d992ef155b0ed4d6b3d8a544a2c76dc0e01e6d00cd4fc59e7e72821fbc9"} Jul 22 09:50:25.540041 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:25.539654 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" event={"ID":"3eacbe5e-a91a-4674-a5ee-f0619022826d","Type":"ContainerStarted","Data":"ca3f837899fbb59382046bdca7c463dd18a30166ade0254b46419f7bae4c48a8"} Jul 22 09:50:25.563735 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:25.563689 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" podStartSLOduration=1.563669918 podStartE2EDuration="1.563669918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:50:25.562288218 +0000 UTC m=+1956.026082659" watchObservedRunningTime="2026-07-22 09:50:25.563669918 +0000 UTC m=+1956.027464355" Jul 22 09:50:25.699186 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:25.699152 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:25.703941 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:25.703917 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:26.545148 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:26.545119 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:26.546285 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:26.546265 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:27.441790 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:27.441752 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:27.518599 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:27.518567 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66"] Jul 22 09:50:27.521937 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:27.521916 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66" Jul 22 09:50:27.526115 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:27.526097 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-876r5\"" Jul 22 09:50:27.526205 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:27.526097 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6285b203-serving-cert\"" Jul 22 09:50:27.527637 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:27.527599 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:27.537101 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:27.537074 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66"] Jul 22 09:50:27.594474 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:27.594446 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:27.609710 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:27.609677 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wcfd\" (UniqueName: \"kubernetes.io/projected/29602c51-c1b8-4549-b288-8eee767a34bd-kube-api-access-8wcfd\") pod \"maas-api-e2e-mt-6285b203-6758665c8f-xcs66\" (UID: \"29602c51-c1b8-4549-b288-8eee767a34bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66" Jul 22 09:50:27.609851 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:27.609747 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29602c51-c1b8-4549-b288-8eee767a34bd-maas-api-tls\") pod \"maas-api-e2e-mt-6285b203-6758665c8f-xcs66\" (UID: \"29602c51-c1b8-4549-b288-8eee767a34bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66" Jul 22 09:50:27.609851 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:27.609839 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29602c51-c1b8-4549-b288-8eee767a34bd-openshift-service-ca\") pod \"maas-api-e2e-mt-6285b203-6758665c8f-xcs66\" (UID: \"29602c51-c1b8-4549-b288-8eee767a34bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66" Jul 22 09:50:27.711210 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:27.711109 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wcfd\" (UniqueName: \"kubernetes.io/projected/29602c51-c1b8-4549-b288-8eee767a34bd-kube-api-access-8wcfd\") pod \"maas-api-e2e-mt-6285b203-6758665c8f-xcs66\" (UID: \"29602c51-c1b8-4549-b288-8eee767a34bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66" Jul 22 09:50:27.711210 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:27.711160 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29602c51-c1b8-4549-b288-8eee767a34bd-maas-api-tls\") pod \"maas-api-e2e-mt-6285b203-6758665c8f-xcs66\" (UID: \"29602c51-c1b8-4549-b288-8eee767a34bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66" Jul 22 09:50:27.711525 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:27.711219 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29602c51-c1b8-4549-b288-8eee767a34bd-openshift-service-ca\") pod \"maas-api-e2e-mt-6285b203-6758665c8f-xcs66\" (UID: \"29602c51-c1b8-4549-b288-8eee767a34bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66" Jul 22 09:50:27.711525 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:50:27.711360 2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-serving-cert: secret "maas-api-e2e-mt-6285b203-serving-cert" not found Jul 22 09:50:27.711525 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:50:27.711434 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29602c51-c1b8-4549-b288-8eee767a34bd-maas-api-tls podName:29602c51-c1b8-4549-b288-8eee767a34bd nodeName:}" failed. No retries permitted until 2026-07-22 09:50:28.211414861 +0000 UTC m=+1958.675209291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/29602c51-c1b8-4549-b288-8eee767a34bd-maas-api-tls") pod "maas-api-e2e-mt-6285b203-6758665c8f-xcs66" (UID: "29602c51-c1b8-4549-b288-8eee767a34bd") : secret "maas-api-e2e-mt-6285b203-serving-cert" not found Jul 22 09:50:27.711829 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:27.711811 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29602c51-c1b8-4549-b288-8eee767a34bd-openshift-service-ca\") pod \"maas-api-e2e-mt-6285b203-6758665c8f-xcs66\" (UID: \"29602c51-c1b8-4549-b288-8eee767a34bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66" Jul 22 09:50:27.720535 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:27.720510 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wcfd\" (UniqueName: \"kubernetes.io/projected/29602c51-c1b8-4549-b288-8eee767a34bd-kube-api-access-8wcfd\") pod \"maas-api-e2e-mt-6285b203-6758665c8f-xcs66\" (UID: \"29602c51-c1b8-4549-b288-8eee767a34bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66" Jul 22 09:50:27.816299 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:27.816250 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:28.153086 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:28.153055 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:28.216218 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:28.216184 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29602c51-c1b8-4549-b288-8eee767a34bd-maas-api-tls\") pod \"maas-api-e2e-mt-6285b203-6758665c8f-xcs66\" (UID: \"29602c51-c1b8-4549-b288-8eee767a34bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66" Jul 22 09:50:28.218510 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:28.218492 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29602c51-c1b8-4549-b288-8eee767a34bd-maas-api-tls\") pod \"maas-api-e2e-mt-6285b203-6758665c8f-xcs66\" (UID: \"29602c51-c1b8-4549-b288-8eee767a34bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66" Jul 22 09:50:28.432759 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:28.432670 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66" Jul 22 09:50:28.480244 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:28.480207 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:28.779900 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:28.779681 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66"] Jul 22 09:50:28.782264 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:50:28.782239 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29602c51_c1b8_4549_b288_8eee767a34bd.slice/crio-c272abc7de51b8144b662a360d98392fa9f0cafe5e3f61f66a5754a85ed17cf0 WatchSource:0}: Error finding container c272abc7de51b8144b662a360d98392fa9f0cafe5e3f61f66a5754a85ed17cf0: Status 404 returned error can't find the container with id c272abc7de51b8144b662a360d98392fa9f0cafe5e3f61f66a5754a85ed17cf0 Jul 22 09:50:28.784128 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:28.784112 2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 09:50:29.563195 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:29.563164 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66" event={"ID":"29602c51-c1b8-4549-b288-8eee767a34bd","Type":"ContainerStarted","Data":"95e7804392f1ac4fa25e4394a7bb49c8c297c4e0b22e6d3fe682eb49a5828017"} Jul 22 09:50:29.563366 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:29.563216 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66" event={"ID":"29602c51-c1b8-4549-b288-8eee767a34bd","Type":"ContainerStarted","Data":"c272abc7de51b8144b662a360d98392fa9f0cafe5e3f61f66a5754a85ed17cf0"} Jul 22 09:50:29.563366 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:29.563266 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66" Jul 22 09:50:29.586534 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:29.586486 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66" podStartSLOduration=2.102106419 podStartE2EDuration="2.586471589s" totalImagesPullingTime="484.36517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:50:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:50:29.585064571 +0000 UTC m=+1960.048859003" watchObservedRunningTime="2026-07-22 09:50:29.586471589 +0000 UTC m=+1960.050266026" Jul 22 09:50:31.996362 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:31.996328 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:35.578441 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:35.578408 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66" Jul 22 09:50:41.185967 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:41.185928 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:50:42.176893 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:42.176337 2549 patch_prober.go:28] interesting pod/e2e-mt-6285b203-openshift-default-df855676c-mz8st container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": read tcp 10.133.0.2:52782->10.133.0.47:15021: read: connection reset by peer" start-of-body= Jul 22 09:50:42.176893 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:42.176423 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" podUID="3eacbe5e-a91a-4674-a5ee-f0619022826d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": read tcp 10.133.0.2:52782->10.133.0.47:15021: read: connection reset by peer" Jul 22 09:50:42.631547 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:42.631513 2549 generic.go:356] "Generic (PLEG): container finished" podID="3eacbe5e-a91a-4674-a5ee-f0619022826d" containerID="1a796d992ef155b0ed4d6b3d8a544a2c76dc0e01e6d00cd4fc59e7e72821fbc9" exitCode=0 Jul 22 09:50:42.631922 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:42.631590 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" event={"ID":"3eacbe5e-a91a-4674-a5ee-f0619022826d","Type":"ContainerDied","Data":"1a796d992ef155b0ed4d6b3d8a544a2c76dc0e01e6d00cd4fc59e7e72821fbc9"} Jul 22 09:50:42.631980 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:42.631966 2549 scope.go:117] "RemoveContainer" containerID="1a796d992ef155b0ed4d6b3d8a544a2c76dc0e01e6d00cd4fc59e7e72821fbc9" Jul 22 09:50:42.632953 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:42.632926 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:50:42.633019 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:42.632994 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:50:42.633057 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:42.633027 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:50:43.639006 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:43.638969 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" event={"ID":"3eacbe5e-a91a-4674-a5ee-f0619022826d","Type":"ContainerStarted","Data":"670a54429ca5113000b27db7c8f28c8cb8e628490265ac7a8ef99ab9d6a32bd8"} Jul 22 09:50:43.698853 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:43.698807 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:43.699065 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:43.699042 2549 patch_prober.go:28] interesting pod/e2e-mt-6285b203-openshift-default-df855676c-mz8st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 22 09:50:43.699114 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:43.699091 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" podUID="3eacbe5e-a91a-4674-a5ee-f0619022826d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 22 09:50:44.699796 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:44.699752 2549 patch_prober.go:28] interesting pod/e2e-mt-6285b203-openshift-default-df855676c-mz8st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 22 09:50:44.700209 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:44.699847 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" podUID="3eacbe5e-a91a-4674-a5ee-f0619022826d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 22 09:50:45.699853 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:45.699818 2549 patch_prober.go:28] interesting pod/e2e-mt-6285b203-openshift-default-df855676c-mz8st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 22 09:50:45.700241 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:45.699875 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" podUID="3eacbe5e-a91a-4674-a5ee-f0619022826d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 22 09:50:46.699070 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:46.699032 2549 patch_prober.go:28] interesting pod/e2e-mt-6285b203-openshift-default-df855676c-mz8st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 22 09:50:46.699249 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:46.699101 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" podUID="3eacbe5e-a91a-4674-a5ee-f0619022826d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 22 09:50:47.702744 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:47.702714 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:47.703218 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:47.703081 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:50:47.703775 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:50:47.703757 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:51:05.228963 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:05.228924 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6285b203-8f2tx"] Jul 22 09:51:05.232311 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:05.232285 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6285b203-8f2tx" Jul 22 09:51:05.234800 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:05.234778 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmn2j\"" Jul 22 09:51:05.242561 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:05.242532 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6285b203-8f2tx"] Jul 22 09:51:05.244992 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:05.244969 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e399df57-5af0-4cb0-915a-efc9fdbeb555-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6285b203-8f2tx\" (UID: \"e399df57-5af0-4cb0-915a-efc9fdbeb555\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6285b203-8f2tx" Jul 22 09:51:05.345910 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:05.345876 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e399df57-5af0-4cb0-915a-efc9fdbeb555-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6285b203-8f2tx\" (UID: \"e399df57-5af0-4cb0-915a-efc9fdbeb555\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6285b203-8f2tx" Jul 22 09:51:05.346519 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:05.346499 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e399df57-5af0-4cb0-915a-efc9fdbeb555-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6285b203-8f2tx\" (UID: \"e399df57-5af0-4cb0-915a-efc9fdbeb555\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6285b203-8f2tx" Jul 22 09:51:05.543392 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:05.543367 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6285b203-8f2tx" Jul 22 09:51:05.694017 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:05.693990 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6285b203-8f2tx"] Jul 22 09:51:05.696516 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:51:05.696484 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode399df57_5af0_4cb0_915a_efc9fdbeb555.slice/crio-a7512cf6806e39aa4e4a17b6f3407350f12bfeebed0214e1814b43bbd5a9a5d6 WatchSource:0}: Error finding container a7512cf6806e39aa4e4a17b6f3407350f12bfeebed0214e1814b43bbd5a9a5d6: Status 404 returned error can't find the container with id a7512cf6806e39aa4e4a17b6f3407350f12bfeebed0214e1814b43bbd5a9a5d6 Jul 22 09:51:05.750341 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:05.750298 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6285b203-8f2tx" event={"ID":"e399df57-5af0-4cb0-915a-efc9fdbeb555","Type":"ContainerStarted","Data":"a7512cf6806e39aa4e4a17b6f3407350f12bfeebed0214e1814b43bbd5a9a5d6"} Jul 22 09:51:06.757111 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:06.757022 2549 generic.go:356] "Generic (PLEG): container finished" podID="e399df57-5af0-4cb0-915a-efc9fdbeb555" containerID="bcdba5f6604f44f657721a1a3ac9d9ffeca276f6959293e315899e975adc6096" exitCode=0 Jul 22 09:51:06.757488 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:06.757108 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6285b203-8f2tx" event={"ID":"e399df57-5af0-4cb0-915a-efc9fdbeb555","Type":"ContainerDied","Data":"bcdba5f6604f44f657721a1a3ac9d9ffeca276f6959293e315899e975adc6096"} Jul 22 09:51:07.893491 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:07.893465 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6285b203-8f2tx" Jul 22 09:51:07.965544 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:07.965513 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e399df57-5af0-4cb0-915a-efc9fdbeb555-maas-api-service-ca\") pod \"e399df57-5af0-4cb0-915a-efc9fdbeb555\" (UID: \"e399df57-5af0-4cb0-915a-efc9fdbeb555\") " Jul 22 09:51:07.965846 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:07.965824 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e399df57-5af0-4cb0-915a-efc9fdbeb555-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e399df57-5af0-4cb0-915a-efc9fdbeb555" (UID: "e399df57-5af0-4cb0-915a-efc9fdbeb555"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:51:08.066568 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:08.066543 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e399df57-5af0-4cb0-915a-efc9fdbeb555-maas-api-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:51:08.768815 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:08.768731 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6285b203-8f2tx" Jul 22 09:51:08.768815 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:08.768770 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6285b203-8f2tx" event={"ID":"e399df57-5af0-4cb0-915a-efc9fdbeb555","Type":"ContainerDied","Data":"a7512cf6806e39aa4e4a17b6f3407350f12bfeebed0214e1814b43bbd5a9a5d6"} Jul 22 09:51:08.768815 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:08.768813 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7512cf6806e39aa4e4a17b6f3407350f12bfeebed0214e1814b43bbd5a9a5d6" Jul 22 09:51:20.865436 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:20.865401 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66"] Jul 22 09:51:20.865957 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:20.865590 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66" podUID="29602c51-c1b8-4549-b288-8eee767a34bd" containerName="maas-api" containerID="cri-o://95e7804392f1ac4fa25e4394a7bb49c8c297c4e0b22e6d3fe682eb49a5828017" gracePeriod=30 Jul 22 09:51:20.875420 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:20.875392 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:51:21.121192 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:21.121137 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66" Jul 22 09:51:21.175428 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:21.175393 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29602c51-c1b8-4549-b288-8eee767a34bd-openshift-service-ca\") pod \"29602c51-c1b8-4549-b288-8eee767a34bd\" (UID: \"29602c51-c1b8-4549-b288-8eee767a34bd\") " Jul 22 09:51:21.175607 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:21.175436 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wcfd\" (UniqueName: \"kubernetes.io/projected/29602c51-c1b8-4549-b288-8eee767a34bd-kube-api-access-8wcfd\") pod \"29602c51-c1b8-4549-b288-8eee767a34bd\" (UID: \"29602c51-c1b8-4549-b288-8eee767a34bd\") " Jul 22 09:51:21.175607 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:21.175463 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29602c51-c1b8-4549-b288-8eee767a34bd-maas-api-tls\") pod \"29602c51-c1b8-4549-b288-8eee767a34bd\" (UID: \"29602c51-c1b8-4549-b288-8eee767a34bd\") " Jul 22 09:51:21.175806 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:21.175782 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29602c51-c1b8-4549-b288-8eee767a34bd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "29602c51-c1b8-4549-b288-8eee767a34bd" (UID: "29602c51-c1b8-4549-b288-8eee767a34bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:51:21.177549 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:21.177518 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29602c51-c1b8-4549-b288-8eee767a34bd-kube-api-access-8wcfd" (OuterVolumeSpecName: "kube-api-access-8wcfd") pod "29602c51-c1b8-4549-b288-8eee767a34bd" (UID: "29602c51-c1b8-4549-b288-8eee767a34bd"). InnerVolumeSpecName "kube-api-access-8wcfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:51:21.177628 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:21.177594 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29602c51-c1b8-4549-b288-8eee767a34bd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "29602c51-c1b8-4549-b288-8eee767a34bd" (UID: "29602c51-c1b8-4549-b288-8eee767a34bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:51:21.276664 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:21.276616 2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29602c51-c1b8-4549-b288-8eee767a34bd-openshift-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:51:21.276664 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:21.276658 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wcfd\" (UniqueName: \"kubernetes.io/projected/29602c51-c1b8-4549-b288-8eee767a34bd-kube-api-access-8wcfd\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:51:21.276664 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:21.276671 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29602c51-c1b8-4549-b288-8eee767a34bd-maas-api-tls\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:51:21.836040 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:21.836002 2549 generic.go:356] "Generic (PLEG): container finished" podID="29602c51-c1b8-4549-b288-8eee767a34bd" containerID="95e7804392f1ac4fa25e4394a7bb49c8c297c4e0b22e6d3fe682eb49a5828017" exitCode=0 Jul 22 09:51:21.836040 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:21.836044 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66" Jul 22 09:51:21.836273 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:21.836093 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66" event={"ID":"29602c51-c1b8-4549-b288-8eee767a34bd","Type":"ContainerDied","Data":"95e7804392f1ac4fa25e4394a7bb49c8c297c4e0b22e6d3fe682eb49a5828017"} Jul 22 09:51:21.836273 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:21.836141 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66" event={"ID":"29602c51-c1b8-4549-b288-8eee767a34bd","Type":"ContainerDied","Data":"c272abc7de51b8144b662a360d98392fa9f0cafe5e3f61f66a5754a85ed17cf0"} Jul 22 09:51:21.836273 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:21.836164 2549 scope.go:117] "RemoveContainer" containerID="95e7804392f1ac4fa25e4394a7bb49c8c297c4e0b22e6d3fe682eb49a5828017" Jul 22 09:51:21.845266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:21.845247 2549 scope.go:117] "RemoveContainer" containerID="95e7804392f1ac4fa25e4394a7bb49c8c297c4e0b22e6d3fe682eb49a5828017" Jul 22 09:51:21.845544 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:51:21.845526 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e7804392f1ac4fa25e4394a7bb49c8c297c4e0b22e6d3fe682eb49a5828017\": container with ID starting with 95e7804392f1ac4fa25e4394a7bb49c8c297c4e0b22e6d3fe682eb49a5828017 not found: ID does not exist" containerID="95e7804392f1ac4fa25e4394a7bb49c8c297c4e0b22e6d3fe682eb49a5828017" Jul 22 09:51:21.845582 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:21.845553 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e7804392f1ac4fa25e4394a7bb49c8c297c4e0b22e6d3fe682eb49a5828017"} err="failed to get container status \"95e7804392f1ac4fa25e4394a7bb49c8c297c4e0b22e6d3fe682eb49a5828017\": rpc error: code = NotFound desc = could not find container \"95e7804392f1ac4fa25e4394a7bb49c8c297c4e0b22e6d3fe682eb49a5828017\": container with ID starting with 95e7804392f1ac4fa25e4394a7bb49c8c297c4e0b22e6d3fe682eb49a5828017 not found: ID does not exist" Jul 22 09:51:21.858393 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:21.858374 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66"] Jul 22 09:51:21.862396 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:21.862374 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6285b203-6758665c8f-xcs66"] Jul 22 09:51:22.340131 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:22.340096 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29602c51-c1b8-4549-b288-8eee767a34bd" path="/var/lib/kubelet/pods/29602c51-c1b8-4549-b288-8eee767a34bd/volumes" Jul 22 09:51:25.317300 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:25.317270 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:51:25.322963 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:25.322937 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6285b203-8f2tx"] Jul 22 09:51:25.326103 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:25.326081 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6285b203-8f2tx"] Jul 22 09:51:26.338962 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:26.338932 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e399df57-5af0-4cb0-915a-efc9fdbeb555" path="/var/lib/kubelet/pods/e399df57-5af0-4cb0-915a-efc9fdbeb555/volumes" Jul 22 09:51:27.658856 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:27.658822 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:51:32.733559 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:32.733522 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:51:50.494176 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:51:50.494141 2549 scope.go:117] "RemoveContainer" containerID="b5fa74fbf21b4945cefb0bb3ab72aba4e148345c67da7f0ded7e7aa361da93fa" Jul 22 09:52:09.259636 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:09.259605 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:10.776262 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:10.776233 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:16.789116 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:16.789027 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:16.903486 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:16.903455 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:17.160618 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:17.160536 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st"] Jul 22 09:52:17.160781 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:17.160759 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" podUID="3eacbe5e-a91a-4674-a5ee-f0619022826d" containerName="istio-proxy" containerID="cri-o://670a54429ca5113000b27db7c8f28c8cb8e628490265ac7a8ef99ab9d6a32bd8" gracePeriod=30 Jul 22 09:52:17.198158 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:17.198134 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:17.469446 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:17.469355 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:17.704684 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:17.704648 2549 patch_prober.go:28] interesting pod/e2e-mt-6285b203-openshift-default-df855676c-mz8st container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 09:52:17.704838 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:17.704718 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" podUID="3eacbe5e-a91a-4674-a5ee-f0619022826d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 09:52:18.544010 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:18.543978 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:18.645799 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:18.645766 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:18.810354 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:18.810249 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:19.600962 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:19.600926 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:19.732748 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:19.732716 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:19.833703 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:19.833673 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:21.300065 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:21.300023 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:22.409734 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.409710 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:52:22.486665 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.486586 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-workload-certs\") pod \"3eacbe5e-a91a-4674-a5ee-f0619022826d\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " Jul 22 09:52:22.486665 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.486642 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-data\") pod \"3eacbe5e-a91a-4674-a5ee-f0619022826d\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " Jul 22 09:52:22.486879 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.486672 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-token\") pod \"3eacbe5e-a91a-4674-a5ee-f0619022826d\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " Jul 22 09:52:22.486879 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.486761 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-workload-socket\") pod \"3eacbe5e-a91a-4674-a5ee-f0619022826d\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " Jul 22 09:52:22.486879 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.486805 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eacbe5e-a91a-4674-a5ee-f0619022826d-istiod-ca-cert\") pod \"3eacbe5e-a91a-4674-a5ee-f0619022826d\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " Jul 22 09:52:22.486879 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.486834 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-envoy\") pod \"3eacbe5e-a91a-4674-a5ee-f0619022826d\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " Jul 22 09:52:22.486879 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.486862 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3eacbe5e-a91a-4674-a5ee-f0619022826d" (UID: "3eacbe5e-a91a-4674-a5ee-f0619022826d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:52:22.487138 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.486873 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-podinfo\") pod \"3eacbe5e-a91a-4674-a5ee-f0619022826d\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " Jul 22 09:52:22.487138 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.486931 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-credential-socket\") pod \"3eacbe5e-a91a-4674-a5ee-f0619022826d\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " Jul 22 09:52:22.487138 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.487020 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ck8xh\" (UniqueName: \"kubernetes.io/projected/3eacbe5e-a91a-4674-a5ee-f0619022826d-kube-api-access-ck8xh\") pod \"3eacbe5e-a91a-4674-a5ee-f0619022826d\" (UID: \"3eacbe5e-a91a-4674-a5ee-f0619022826d\") " Jul 22 09:52:22.487320 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.487234 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eacbe5e-a91a-4674-a5ee-f0619022826d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3eacbe5e-a91a-4674-a5ee-f0619022826d" (UID: "3eacbe5e-a91a-4674-a5ee-f0619022826d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:52:22.487383 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.487349 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:52:22.487383 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.487368 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eacbe5e-a91a-4674-a5ee-f0619022826d-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:52:22.487383 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.487364 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3eacbe5e-a91a-4674-a5ee-f0619022826d" (UID: "3eacbe5e-a91a-4674-a5ee-f0619022826d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:52:22.487549 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.487512 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3eacbe5e-a91a-4674-a5ee-f0619022826d" (UID: "3eacbe5e-a91a-4674-a5ee-f0619022826d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:52:22.487641 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.487619 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-data" (OuterVolumeSpecName: "istio-data") pod "3eacbe5e-a91a-4674-a5ee-f0619022826d" (UID: "3eacbe5e-a91a-4674-a5ee-f0619022826d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:52:22.489032 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.489005 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-token" (OuterVolumeSpecName: "istio-token") pod "3eacbe5e-a91a-4674-a5ee-f0619022826d" (UID: "3eacbe5e-a91a-4674-a5ee-f0619022826d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:52:22.489167 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.489117 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eacbe5e-a91a-4674-a5ee-f0619022826d-kube-api-access-ck8xh" (OuterVolumeSpecName: "kube-api-access-ck8xh") pod "3eacbe5e-a91a-4674-a5ee-f0619022826d" (UID: "3eacbe5e-a91a-4674-a5ee-f0619022826d"). InnerVolumeSpecName "kube-api-access-ck8xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:52:22.489299 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.489270 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3eacbe5e-a91a-4674-a5ee-f0619022826d" (UID: "3eacbe5e-a91a-4674-a5ee-f0619022826d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:52:22.489425 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.489346 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3eacbe5e-a91a-4674-a5ee-f0619022826d" (UID: "3eacbe5e-a91a-4674-a5ee-f0619022826d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 09:52:22.588538 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.588504 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:52:22.588538 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.588533 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:52:22.588538 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.588544 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck8xh\" (UniqueName: \"kubernetes.io/projected/3eacbe5e-a91a-4674-a5ee-f0619022826d-kube-api-access-ck8xh\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:52:22.588772 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.588552 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:52:22.588772 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.588560 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:52:22.588772 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.588568 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:52:22.588772 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:22.588576 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eacbe5e-a91a-4674-a5ee-f0619022826d-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:52:23.138417 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:23.138384 2549 generic.go:356] "Generic (PLEG): container finished" podID="3eacbe5e-a91a-4674-a5ee-f0619022826d" containerID="670a54429ca5113000b27db7c8f28c8cb8e628490265ac7a8ef99ab9d6a32bd8" exitCode=0 Jul 22 09:52:23.138620 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:23.138439 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" Jul 22 09:52:23.138620 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:23.138470 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" event={"ID":"3eacbe5e-a91a-4674-a5ee-f0619022826d","Type":"ContainerDied","Data":"670a54429ca5113000b27db7c8f28c8cb8e628490265ac7a8ef99ab9d6a32bd8"} Jul 22 09:52:23.138620 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:23.138523 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st" event={"ID":"3eacbe5e-a91a-4674-a5ee-f0619022826d","Type":"ContainerDied","Data":"ca3f837899fbb59382046bdca7c463dd18a30166ade0254b46419f7bae4c48a8"} Jul 22 09:52:23.138620 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:23.138547 2549 scope.go:117] "RemoveContainer" containerID="670a54429ca5113000b27db7c8f28c8cb8e628490265ac7a8ef99ab9d6a32bd8" Jul 22 09:52:23.149994 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:23.149977 2549 scope.go:117] "RemoveContainer" containerID="1a796d992ef155b0ed4d6b3d8a544a2c76dc0e01e6d00cd4fc59e7e72821fbc9" Jul 22 09:52:23.157813 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:23.157797 2549 scope.go:117] "RemoveContainer" containerID="670a54429ca5113000b27db7c8f28c8cb8e628490265ac7a8ef99ab9d6a32bd8" Jul 22 09:52:23.158051 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:52:23.158034 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"670a54429ca5113000b27db7c8f28c8cb8e628490265ac7a8ef99ab9d6a32bd8\": container with ID starting with 670a54429ca5113000b27db7c8f28c8cb8e628490265ac7a8ef99ab9d6a32bd8 not found: ID does not exist" containerID="670a54429ca5113000b27db7c8f28c8cb8e628490265ac7a8ef99ab9d6a32bd8" Jul 22 09:52:23.158101 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:23.158058 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"670a54429ca5113000b27db7c8f28c8cb8e628490265ac7a8ef99ab9d6a32bd8"} err="failed to get container status \"670a54429ca5113000b27db7c8f28c8cb8e628490265ac7a8ef99ab9d6a32bd8\": rpc error: code = NotFound desc = could not find container \"670a54429ca5113000b27db7c8f28c8cb8e628490265ac7a8ef99ab9d6a32bd8\": container with ID starting with 670a54429ca5113000b27db7c8f28c8cb8e628490265ac7a8ef99ab9d6a32bd8 not found: ID does not exist" Jul 22 09:52:23.158101 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:23.158073 2549 scope.go:117] "RemoveContainer" containerID="1a796d992ef155b0ed4d6b3d8a544a2c76dc0e01e6d00cd4fc59e7e72821fbc9" Jul 22 09:52:23.158286 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:52:23.158268 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a796d992ef155b0ed4d6b3d8a544a2c76dc0e01e6d00cd4fc59e7e72821fbc9\": container with ID starting with 1a796d992ef155b0ed4d6b3d8a544a2c76dc0e01e6d00cd4fc59e7e72821fbc9 not found: ID does not exist" containerID="1a796d992ef155b0ed4d6b3d8a544a2c76dc0e01e6d00cd4fc59e7e72821fbc9" Jul 22 09:52:23.158363 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:23.158296 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a796d992ef155b0ed4d6b3d8a544a2c76dc0e01e6d00cd4fc59e7e72821fbc9"} err="failed to get container status \"1a796d992ef155b0ed4d6b3d8a544a2c76dc0e01e6d00cd4fc59e7e72821fbc9\": rpc error: code = NotFound desc = could not find container \"1a796d992ef155b0ed4d6b3d8a544a2c76dc0e01e6d00cd4fc59e7e72821fbc9\": container with ID starting with 1a796d992ef155b0ed4d6b3d8a544a2c76dc0e01e6d00cd4fc59e7e72821fbc9 not found: ID does not exist" Jul 22 09:52:23.163988 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:23.163964 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st"] Jul 22 09:52:23.171450 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:23.171430 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6285b203-openshift-default-df855676c-mz8st"] Jul 22 09:52:24.340772 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:24.340741 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eacbe5e-a91a-4674-a5ee-f0619022826d" path="/var/lib/kubelet/pods/3eacbe5e-a91a-4674-a5ee-f0619022826d/volumes" Jul 22 09:52:27.671367 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:27.671323 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:27.782173 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:27.782143 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:28.424285 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:28.424250 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:32.678024 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:32.677989 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:32.785730 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:32.785698 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:35.406122 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:35.406087 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:35.473909 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:35.473879 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:35.591542 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:35.591510 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:35.724627 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:35.724543 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:46.211923 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:46.211883 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:47.156065 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:47.156029 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:52:50.412293 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:50.412264 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcsgw_dc8d8789-74a0-44ec-8902-44a9650c41ac/ovn-acl-logging/0.log" Jul 22 09:52:50.432749 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:52:50.432730 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcsgw_dc8d8789-74a0-44ec-8902-44a9650c41ac/ovn-acl-logging/0.log" Jul 22 09:53:07.589938 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:07.589899 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:10.164972 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:10.164920 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:14.657491 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:14.657435 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:14.781939 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:14.781911 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:15.074807 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.074773 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:15.362133 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.362038 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx"] Jul 22 09:53:15.362501 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.362480 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e399df57-5af0-4cb0-915a-efc9fdbeb555" containerName="revoke-keys" Jul 22 09:53:15.362501 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.362503 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e399df57-5af0-4cb0-915a-efc9fdbeb555" containerName="revoke-keys" Jul 22 09:53:15.362703 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.362517 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29602c51-c1b8-4549-b288-8eee767a34bd" containerName="maas-api" Jul 22 09:53:15.362703 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.362526 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="29602c51-c1b8-4549-b288-8eee767a34bd" containerName="maas-api" Jul 22 09:53:15.362703 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.362556 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eacbe5e-a91a-4674-a5ee-f0619022826d" containerName="istio-proxy" Jul 22 09:53:15.362703 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.362566 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eacbe5e-a91a-4674-a5ee-f0619022826d" containerName="istio-proxy" Jul 22 09:53:15.362703 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.362578 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eacbe5e-a91a-4674-a5ee-f0619022826d" containerName="istio-proxy" Jul 22 09:53:15.362703 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.362586 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eacbe5e-a91a-4674-a5ee-f0619022826d" containerName="istio-proxy" Jul 22 09:53:15.362703 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.362665 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eacbe5e-a91a-4674-a5ee-f0619022826d" containerName="istio-proxy" Jul 22 09:53:15.362703 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.362678 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eacbe5e-a91a-4674-a5ee-f0619022826d" containerName="istio-proxy" Jul 22 09:53:15.362703 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.362688 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="e399df57-5af0-4cb0-915a-efc9fdbeb555" containerName="revoke-keys" Jul 22 09:53:15.362703 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.362696 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="29602c51-c1b8-4549-b288-8eee767a34bd" containerName="maas-api" Jul 22 09:53:15.365884 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.365862 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.368499 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.368480 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-982fc1-openshift-default-dockercfg-glxsj\"" Jul 22 09:53:15.380006 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.379986 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx"] Jul 22 09:53:15.394300 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.394278 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:15.435593 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.435566 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-workload-socket\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.435702 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.435595 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-data\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.435702 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.435619 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-token\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.435702 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.435667 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpjvf\" (UniqueName: \"kubernetes.io/projected/8e2ec773-7540-4012-8293-ec4fd24a7606-kube-api-access-wpjvf\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.435809 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.435722 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-podinfo\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.435809 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.435749 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2ec773-7540-4012-8293-ec4fd24a7606-istiod-ca-cert\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.435809 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.435794 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-workload-certs\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.435905 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.435841 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-credential-socket\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.435905 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.435865 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-envoy\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.537057 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.537020 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-workload-certs\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.537250 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.537078 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-credential-socket\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.537250 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.537101 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-envoy\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.537250 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.537227 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-workload-socket\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.537467 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.537268 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-data\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.537467 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.537345 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-token\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.537467 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.537382 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpjvf\" (UniqueName: \"kubernetes.io/projected/8e2ec773-7540-4012-8293-ec4fd24a7606-kube-api-access-wpjvf\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.537467 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.537435 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-workload-certs\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.537467 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.537441 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-podinfo\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.537726 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.537496 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2ec773-7540-4012-8293-ec4fd24a7606-istiod-ca-cert\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.537726 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.537535 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-credential-socket\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.537726 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.537621 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-workload-socket\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.537726 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.537639 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-data\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.538074 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.538047 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2ec773-7540-4012-8293-ec4fd24a7606-istiod-ca-cert\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.539401 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.539382 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-envoy\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.539674 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.539658 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-podinfo\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.548621 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.548597 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-token\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.548756 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.548742 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpjvf\" (UniqueName: \"kubernetes.io/projected/8e2ec773-7540-4012-8293-ec4fd24a7606-kube-api-access-wpjvf\") pod \"e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.641499 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.641428 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:15.677536 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.677510 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:15.818475 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.818448 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx"] Jul 22 09:53:15.820540 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:53:15.820505 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2ec773_7540_4012_8293_ec4fd24a7606.slice/crio-023a214f801043b4ad39cd7d8c5fd21252e2e9d21f9e4c5ef8d8e637ccc5bb08 WatchSource:0}: Error finding container 023a214f801043b4ad39cd7d8c5fd21252e2e9d21f9e4c5ef8d8e637ccc5bb08: Status 404 returned error can't find the container with id 023a214f801043b4ad39cd7d8c5fd21252e2e9d21f9e4c5ef8d8e637ccc5bb08 Jul 22 09:53:15.822581 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.822548 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:53:15.822683 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.822616 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:53:15.822683 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:15.822642 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:53:16.398424 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:16.398387 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" event={"ID":"8e2ec773-7540-4012-8293-ec4fd24a7606","Type":"ContainerStarted","Data":"9f35d9a4a91fcccae8e9b38cbb71f4b55843a86119b6d7433b57f0d23a3d1556"} Jul 22 09:53:16.398648 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:16.398441 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" event={"ID":"8e2ec773-7540-4012-8293-ec4fd24a7606","Type":"ContainerStarted","Data":"023a214f801043b4ad39cd7d8c5fd21252e2e9d21f9e4c5ef8d8e637ccc5bb08"} Jul 22 09:53:16.424161 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:16.424111 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" podStartSLOduration=1.424092001 podStartE2EDuration="1.424092001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:53:16.419910423 +0000 UTC m=+2126.883704862" watchObservedRunningTime="2026-07-22 09:53:16.424092001 +0000 UTC m=+2126.887886443" Jul 22 09:53:16.677744 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:16.677648 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:16.682767 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:16.682744 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:17.403925 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:17.403891 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:17.404864 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:17.404843 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:18.448596 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:18.448560 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:18.547784 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:18.547744 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:18.677423 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:18.677393 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g"] Jul 22 09:53:18.680791 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:18.680775 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g" Jul 22 09:53:18.683414 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:18.683397 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-982fc1-serving-cert\"" Jul 22 09:53:18.683707 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:18.683692 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-876r5\"" Jul 22 09:53:18.705279 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:18.705209 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g"] Jul 22 09:53:18.719836 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:18.719795 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:18.765282 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:18.765246 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e22d96-c8c6-4567-842a-5b511fedb361-openshift-service-ca\") pod \"maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g\" (UID: \"b7e22d96-c8c6-4567-842a-5b511fedb361\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g" Jul 22 09:53:18.765462 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:18.765287 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7e22d96-c8c6-4567-842a-5b511fedb361-maas-api-tls\") pod \"maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g\" (UID: \"b7e22d96-c8c6-4567-842a-5b511fedb361\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g" Jul 22 09:53:18.765462 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:18.765429 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7xwj\" (UniqueName: \"kubernetes.io/projected/b7e22d96-c8c6-4567-842a-5b511fedb361-kube-api-access-z7xwj\") pod \"maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g\" (UID: \"b7e22d96-c8c6-4567-842a-5b511fedb361\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g" Jul 22 09:53:18.866702 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:18.866670 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7e22d96-c8c6-4567-842a-5b511fedb361-maas-api-tls\") pod \"maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g\" (UID: \"b7e22d96-c8c6-4567-842a-5b511fedb361\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g" Jul 22 09:53:18.866887 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:18.866776 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7xwj\" (UniqueName: \"kubernetes.io/projected/b7e22d96-c8c6-4567-842a-5b511fedb361-kube-api-access-z7xwj\") pod \"maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g\" (UID: \"b7e22d96-c8c6-4567-842a-5b511fedb361\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g" Jul 22 09:53:18.866887 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:18.866809 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e22d96-c8c6-4567-842a-5b511fedb361-openshift-service-ca\") pod \"maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g\" (UID: \"b7e22d96-c8c6-4567-842a-5b511fedb361\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g" Jul 22 09:53:18.867465 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:18.867441 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e22d96-c8c6-4567-842a-5b511fedb361-openshift-service-ca\") pod \"maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g\" (UID: \"b7e22d96-c8c6-4567-842a-5b511fedb361\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g" Jul 22 09:53:18.869088 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:18.869070 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7e22d96-c8c6-4567-842a-5b511fedb361-maas-api-tls\") pod \"maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g\" (UID: \"b7e22d96-c8c6-4567-842a-5b511fedb361\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g" Jul 22 09:53:18.877464 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:18.877441 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7xwj\" (UniqueName: \"kubernetes.io/projected/b7e22d96-c8c6-4567-842a-5b511fedb361-kube-api-access-z7xwj\") pod \"maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g\" (UID: \"b7e22d96-c8c6-4567-842a-5b511fedb361\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g" Jul 22 09:53:18.956020 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:18.955940 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:18.991158 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:18.991135 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g" Jul 22 09:53:19.130119 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:19.130086 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g"] Jul 22 09:53:19.131502 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:53:19.131477 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e22d96_c8c6_4567_842a_5b511fedb361.slice/crio-045ed9fb954c93f4a6e5b667ec62c9276d19608d2d603e91d1f04667ac48c6e0 WatchSource:0}: Error finding container 045ed9fb954c93f4a6e5b667ec62c9276d19608d2d603e91d1f04667ac48c6e0: Status 404 returned error can't find the container with id 045ed9fb954c93f4a6e5b667ec62c9276d19608d2d603e91d1f04667ac48c6e0 Jul 22 09:53:19.415338 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:19.415285 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g" event={"ID":"b7e22d96-c8c6-4567-842a-5b511fedb361","Type":"ContainerStarted","Data":"045ed9fb954c93f4a6e5b667ec62c9276d19608d2d603e91d1f04667ac48c6e0"} Jul 22 09:53:19.459794 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:19.459761 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:19.562011 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:19.561968 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:20.421820 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:20.421789 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g" event={"ID":"b7e22d96-c8c6-4567-842a-5b511fedb361","Type":"ContainerStarted","Data":"5abaff829723b3fbc09e2043ee6d92792dc5ff5aacf71e9d256ccbdbf3622b59"} Jul 22 09:53:20.422000 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:20.421851 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g" Jul 22 09:53:20.442334 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:20.442267 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g" podStartSLOduration=1.998145099 podStartE2EDuration="2.442251779s" totalImagesPullingTime="444.10668ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:53:20.441298068 +0000 UTC m=+2130.905092533" watchObservedRunningTime="2026-07-22 09:53:20.442251779 +0000 UTC m=+2130.906046250" Jul 22 09:53:22.956977 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:22.956944 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:23.334395 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.334343 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:23.604555 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.604465 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6"] Jul 22 09:53:23.607934 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.607913 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.610677 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.610651 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b1d31b-openshift-default-dockercfg-nhd8z\"" Jul 22 09:53:23.625150 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.625126 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6"] Jul 22 09:53:23.636391 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.636363 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:23.705650 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.705618 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb9jn\" (UniqueName: \"kubernetes.io/projected/7f518bf2-4686-487b-acd7-9eaa0257ba50-kube-api-access-rb9jn\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.705823 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.705657 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-data\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.705823 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.705696 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-workload-certs\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.705823 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.705720 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f518bf2-4686-487b-acd7-9eaa0257ba50-istiod-ca-cert\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.705823 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.705755 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-credential-socket\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.705823 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.705776 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-token\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.706031 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.705866 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-workload-socket\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.706031 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.705944 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-envoy\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.706031 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.705962 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-podinfo\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.806855 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.806818 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-envoy\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.806855 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.806855 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-podinfo\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.807121 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.806895 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb9jn\" (UniqueName: \"kubernetes.io/projected/7f518bf2-4686-487b-acd7-9eaa0257ba50-kube-api-access-rb9jn\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.807121 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.806921 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-data\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.807121 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.806958 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-workload-certs\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.807121 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.806981 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f518bf2-4686-487b-acd7-9eaa0257ba50-istiod-ca-cert\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.807121 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.807032 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-credential-socket\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.807121 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.807055 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-token\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.807121 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.807090 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-workload-socket\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.807483 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.807365 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-data\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.807531 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.807492 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-credential-socket\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.807584 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.807533 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-workload-socket\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.807770 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.807751 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f518bf2-4686-487b-acd7-9eaa0257ba50-istiod-ca-cert\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.807812 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.807771 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-workload-certs\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.809228 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.809212 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-envoy\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.809355 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.809264 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-podinfo\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.821431 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.821407 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-token\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.822500 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.822475 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb9jn\" (UniqueName: \"kubernetes.io/projected/7f518bf2-4686-487b-acd7-9eaa0257ba50-kube-api-access-rb9jn\") pod \"e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:23.918365 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:23.918274 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:24.051988 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:24.051958 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6"] Jul 22 09:53:24.052959 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:53:24.052929 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f518bf2_4686_487b_acd7_9eaa0257ba50.slice/crio-56f12077cd8dea5bc7be2085dc2c95eba863e9a065bf9f2d9793a6bb1db71dd5 WatchSource:0}: Error finding container 56f12077cd8dea5bc7be2085dc2c95eba863e9a065bf9f2d9793a6bb1db71dd5: Status 404 returned error can't find the container with id 56f12077cd8dea5bc7be2085dc2c95eba863e9a065bf9f2d9793a6bb1db71dd5 Jul 22 09:53:24.055060 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:24.055029 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:53:24.055118 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:24.055094 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:53:24.055162 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:24.055122 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:53:24.300915 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:24.300881 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:24.449118 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:24.449089 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" event={"ID":"7f518bf2-4686-487b-acd7-9eaa0257ba50","Type":"ContainerStarted","Data":"8ed8fd404de3865721b5efcd97101b9d5e747effbef7ad67e1bb244a041fd608"} Jul 22 09:53:24.449295 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:24.449136 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" event={"ID":"7f518bf2-4686-487b-acd7-9eaa0257ba50","Type":"ContainerStarted","Data":"56f12077cd8dea5bc7be2085dc2c95eba863e9a065bf9f2d9793a6bb1db71dd5"} Jul 22 09:53:24.472731 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:24.472686 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" podStartSLOduration=1.47267236 podStartE2EDuration="1.47267236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:53:24.469999819 +0000 UTC m=+2134.933794257" watchObservedRunningTime="2026-07-22 09:53:24.47267236 +0000 UTC m=+2134.936466798" Jul 22 09:53:24.918407 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:24.918367 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:25.923198 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:25.923167 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:26.439832 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:26.439807 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g" Jul 22 09:53:26.459461 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:26.459420 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:26.460275 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:26.460258 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:26.471719 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:26.471688 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:26.579331 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:26.562349 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb"] Jul 22 09:53:26.582282 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:26.582250 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:26.593527 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:26.593498 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" Jul 22 09:53:26.593940 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:26.593914 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb"] Jul 22 09:53:26.596386 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:26.596354 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b1d31b-serving-cert\"" Jul 22 09:53:26.730793 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:26.730703 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d6a8929-b466-46ed-8b70-c0063524738d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb\" (UID: \"2d6a8929-b466-46ed-8b70-c0063524738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" Jul 22 09:53:26.730793 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:26.730772 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l646\" (UniqueName: \"kubernetes.io/projected/2d6a8929-b466-46ed-8b70-c0063524738d-kube-api-access-8l646\") pod \"maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb\" (UID: \"2d6a8929-b466-46ed-8b70-c0063524738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" Jul 22 09:53:26.730981 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:26.730799 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d6a8929-b466-46ed-8b70-c0063524738d-maas-api-tls\") pod \"maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb\" (UID: \"2d6a8929-b466-46ed-8b70-c0063524738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" Jul 22 09:53:26.832188 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:26.832153 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l646\" (UniqueName: \"kubernetes.io/projected/2d6a8929-b466-46ed-8b70-c0063524738d-kube-api-access-8l646\") pod \"maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb\" (UID: \"2d6a8929-b466-46ed-8b70-c0063524738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" Jul 22 09:53:26.832359 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:26.832196 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d6a8929-b466-46ed-8b70-c0063524738d-maas-api-tls\") pod \"maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb\" (UID: \"2d6a8929-b466-46ed-8b70-c0063524738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" Jul 22 09:53:26.832359 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:26.832244 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d6a8929-b466-46ed-8b70-c0063524738d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb\" (UID: \"2d6a8929-b466-46ed-8b70-c0063524738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" Jul 22 09:53:26.832467 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:53:26.832384 2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-serving-cert: secret "maas-api-e2e-inf-b-b1d31b-serving-cert" not found Jul 22 09:53:26.832467 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:53:26.832462 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d6a8929-b466-46ed-8b70-c0063524738d-maas-api-tls podName:2d6a8929-b466-46ed-8b70-c0063524738d nodeName:}" failed. No retries permitted until 2026-07-22 09:53:27.332443055 +0000 UTC m=+2137.796237471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d6a8929-b466-46ed-8b70-c0063524738d-maas-api-tls") pod "maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" (UID: "2d6a8929-b466-46ed-8b70-c0063524738d") : secret "maas-api-e2e-inf-b-b1d31b-serving-cert" not found Jul 22 09:53:26.832847 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:26.832830 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d6a8929-b466-46ed-8b70-c0063524738d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb\" (UID: \"2d6a8929-b466-46ed-8b70-c0063524738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" Jul 22 09:53:26.841472 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:26.841445 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l646\" (UniqueName: \"kubernetes.io/projected/2d6a8929-b466-46ed-8b70-c0063524738d-kube-api-access-8l646\") pod \"maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb\" (UID: \"2d6a8929-b466-46ed-8b70-c0063524738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" Jul 22 09:53:27.143182 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:27.143118 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:27.243087 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:27.243040 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:27.337527 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:27.337497 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d6a8929-b466-46ed-8b70-c0063524738d-maas-api-tls\") pod \"maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb\" (UID: \"2d6a8929-b466-46ed-8b70-c0063524738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" Jul 22 09:53:27.339894 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:27.339867 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d6a8929-b466-46ed-8b70-c0063524738d-maas-api-tls\") pod \"maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb\" (UID: \"2d6a8929-b466-46ed-8b70-c0063524738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" Jul 22 09:53:27.505146 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:27.505056 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" Jul 22 09:53:27.661394 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:27.661345 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb"] Jul 22 09:53:27.683875 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:53:27.683818 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6a8929_b466_46ed_8b70_c0063524738d.slice/crio-319e17ff61ba1fb9dac3db7644ac20df0ea27a26861c004776e1029ca5ec6321 WatchSource:0}: Error finding container 319e17ff61ba1fb9dac3db7644ac20df0ea27a26861c004776e1029ca5ec6321: Status 404 returned error can't find the container with id 319e17ff61ba1fb9dac3db7644ac20df0ea27a26861c004776e1029ca5ec6321 Jul 22 09:53:28.498001 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:28.497904 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" event={"ID":"2d6a8929-b466-46ed-8b70-c0063524738d","Type":"ContainerStarted","Data":"405a5bdd9d7892e772c955008481302038f7e1313e55c3ec861720a6510d68f1"} Jul 22 09:53:28.498001 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:28.497969 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" event={"ID":"2d6a8929-b466-46ed-8b70-c0063524738d","Type":"ContainerStarted","Data":"319e17ff61ba1fb9dac3db7644ac20df0ea27a26861c004776e1029ca5ec6321"} Jul 22 09:53:28.498523 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:28.498213 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" Jul 22 09:53:28.521954 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:28.521899 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" podStartSLOduration=1.99675489 podStartE2EDuration="2.521883134s" totalImagesPullingTime="525.128244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:53:28.51927881 +0000 UTC m=+2138.983073249" watchObservedRunningTime="2026-07-22 09:53:28.521883134 +0000 UTC m=+2138.985677615" Jul 22 09:53:31.033277 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:31.033235 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:32.406183 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:32.406075 2549 patch_prober.go:28] interesting pod/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 22 09:53:32.406183 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:32.406136 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" podUID="8e2ec773-7540-4012-8293-ec4fd24a7606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 22 09:53:32.522763 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:32.522732 2549 generic.go:356] "Generic (PLEG): container finished" podID="8e2ec773-7540-4012-8293-ec4fd24a7606" containerID="9f35d9a4a91fcccae8e9b38cbb71f4b55843a86119b6d7433b57f0d23a3d1556" exitCode=0 Jul 22 09:53:32.522934 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:32.522806 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" event={"ID":"8e2ec773-7540-4012-8293-ec4fd24a7606","Type":"ContainerDied","Data":"9f35d9a4a91fcccae8e9b38cbb71f4b55843a86119b6d7433b57f0d23a3d1556"} Jul 22 09:53:32.523210 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:32.523197 2549 scope.go:117] "RemoveContainer" containerID="9f35d9a4a91fcccae8e9b38cbb71f4b55843a86119b6d7433b57f0d23a3d1556" Jul 22 09:53:32.524225 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:32.524194 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:53:32.524343 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:32.524295 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:53:32.524427 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:32.524377 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:53:33.530048 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:33.530013 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" event={"ID":"8e2ec773-7540-4012-8293-ec4fd24a7606","Type":"ContainerStarted","Data":"2f67c69abaddab4614b76104f5e8ac03d4004f41a91e9d3f22e2349372715745"} Jul 22 09:53:33.678493 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:33.678452 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:33.678711 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:33.678511 2549 patch_prober.go:28] interesting pod/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 22 09:53:33.678711 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:33.678574 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" podUID="8e2ec773-7540-4012-8293-ec4fd24a7606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 22 09:53:34.678918 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:34.678876 2549 patch_prober.go:28] interesting pod/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 22 09:53:34.679516 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:34.679480 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" podUID="8e2ec773-7540-4012-8293-ec4fd24a7606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 22 09:53:35.517553 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:35.517524 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" Jul 22 09:53:35.677947 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:35.677912 2549 patch_prober.go:28] interesting pod/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 22 09:53:35.678233 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:35.678193 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" podUID="8e2ec773-7540-4012-8293-ec4fd24a7606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 22 09:53:36.678460 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:36.677960 2549 patch_prober.go:28] interesting pod/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 22 09:53:36.678460 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:36.678030 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" podUID="8e2ec773-7540-4012-8293-ec4fd24a7606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 22 09:53:37.682699 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:37.682670 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:37.683046 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:37.682928 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:37.683730 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:37.683713 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:53:38.555662 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:38.555627 2549 generic.go:356] "Generic (PLEG): container finished" podID="7f518bf2-4686-487b-acd7-9eaa0257ba50" containerID="8ed8fd404de3865721b5efcd97101b9d5e747effbef7ad67e1bb244a041fd608" exitCode=0 Jul 22 09:53:38.555912 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:38.555707 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" event={"ID":"7f518bf2-4686-487b-acd7-9eaa0257ba50","Type":"ContainerDied","Data":"8ed8fd404de3865721b5efcd97101b9d5e747effbef7ad67e1bb244a041fd608"} Jul 22 09:53:38.556217 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:38.556202 2549 scope.go:117] "RemoveContainer" containerID="8ed8fd404de3865721b5efcd97101b9d5e747effbef7ad67e1bb244a041fd608" Jul 22 09:53:38.557137 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:38.557111 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:53:38.557194 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:38.557175 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:53:38.557229 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:38.557205 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:53:39.562853 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:39.562815 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" event={"ID":"7f518bf2-4686-487b-acd7-9eaa0257ba50","Type":"ContainerStarted","Data":"0b188fd954f255de96b4872b6f75248353995e1f4e720c9a7c3d7c2895a011c4"} Jul 22 09:53:39.919629 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:39.919467 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:39.920961 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:39.920933 2549 patch_prober.go:28] interesting pod/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 22 09:53:39.921462 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:39.921157 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" podUID="7f518bf2-4686-487b-acd7-9eaa0257ba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 22 09:53:40.519669 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:40.519641 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:40.918981 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:40.918948 2549 patch_prober.go:28] interesting pod/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 22 09:53:40.919383 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:40.919012 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" podUID="7f518bf2-4686-487b-acd7-9eaa0257ba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 22 09:53:41.415051 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:41.415011 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:41.459347 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:41.459295 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:41.541168 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:41.541130 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:41.918555 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:41.918524 2549 patch_prober.go:28] interesting pod/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 22 09:53:41.918726 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:41.918575 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" podUID="7f518bf2-4686-487b-acd7-9eaa0257ba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 22 09:53:42.918961 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:42.918926 2549 patch_prober.go:28] interesting pod/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 22 09:53:42.919361 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:42.918979 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" podUID="7f518bf2-4686-487b-acd7-9eaa0257ba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 22 09:53:43.918648 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:43.918615 2549 patch_prober.go:28] interesting pod/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 22 09:53:43.918820 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:43.918675 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" podUID="7f518bf2-4686-487b-acd7-9eaa0257ba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 22 09:53:44.923279 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:44.923248 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:44.924198 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:44.924180 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:53:48.121930 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:48.121900 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:48.226784 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:48.226751 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:53:57.180500 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:53:57.180445 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:54:22.929228 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:54:22.929188 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:54:23.070722 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:54:23.070692 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:54:34.285943 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:54:34.285913 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:01.089113 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:01.089077 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:04.695960 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:04.695921 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:12.664249 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:12.664163 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g"] Jul 22 09:55:12.664815 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:12.664457 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g" podUID="b7e22d96-c8c6-4567-842a-5b511fedb361" containerName="maas-api" containerID="cri-o://5abaff829723b3fbc09e2043ee6d92792dc5ff5aacf71e9d256ccbdbf3622b59" gracePeriod=30 Jul 22 09:55:12.686323 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:12.686281 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:12.912719 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:12.912697 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g" Jul 22 09:55:12.987197 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:12.987127 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7xwj\" (UniqueName: \"kubernetes.io/projected/b7e22d96-c8c6-4567-842a-5b511fedb361-kube-api-access-z7xwj\") pod \"b7e22d96-c8c6-4567-842a-5b511fedb361\" (UID: \"b7e22d96-c8c6-4567-842a-5b511fedb361\") " Jul 22 09:55:12.987197 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:12.987187 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7e22d96-c8c6-4567-842a-5b511fedb361-maas-api-tls\") pod \"b7e22d96-c8c6-4567-842a-5b511fedb361\" (UID: \"b7e22d96-c8c6-4567-842a-5b511fedb361\") " Jul 22 09:55:12.987425 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:12.987246 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e22d96-c8c6-4567-842a-5b511fedb361-openshift-service-ca\") pod \"b7e22d96-c8c6-4567-842a-5b511fedb361\" (UID: \"b7e22d96-c8c6-4567-842a-5b511fedb361\") " Jul 22 09:55:12.987632 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:12.987607 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7e22d96-c8c6-4567-842a-5b511fedb361-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b7e22d96-c8c6-4567-842a-5b511fedb361" (UID: "b7e22d96-c8c6-4567-842a-5b511fedb361"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:55:12.989357 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:12.989323 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7e22d96-c8c6-4567-842a-5b511fedb361-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b7e22d96-c8c6-4567-842a-5b511fedb361" (UID: "b7e22d96-c8c6-4567-842a-5b511fedb361"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:55:12.989459 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:12.989361 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7e22d96-c8c6-4567-842a-5b511fedb361-kube-api-access-z7xwj" (OuterVolumeSpecName: "kube-api-access-z7xwj") pod "b7e22d96-c8c6-4567-842a-5b511fedb361" (UID: "b7e22d96-c8c6-4567-842a-5b511fedb361"). InnerVolumeSpecName "kube-api-access-z7xwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:55:13.036401 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:13.036366 2549 generic.go:356] "Generic (PLEG): container finished" podID="b7e22d96-c8c6-4567-842a-5b511fedb361" containerID="5abaff829723b3fbc09e2043ee6d92792dc5ff5aacf71e9d256ccbdbf3622b59" exitCode=0 Jul 22 09:55:13.036513 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:13.036419 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g" Jul 22 09:55:13.036513 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:13.036461 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g" event={"ID":"b7e22d96-c8c6-4567-842a-5b511fedb361","Type":"ContainerDied","Data":"5abaff829723b3fbc09e2043ee6d92792dc5ff5aacf71e9d256ccbdbf3622b59"} Jul 22 09:55:13.036513 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:13.036512 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g" event={"ID":"b7e22d96-c8c6-4567-842a-5b511fedb361","Type":"ContainerDied","Data":"045ed9fb954c93f4a6e5b667ec62c9276d19608d2d603e91d1f04667ac48c6e0"} Jul 22 09:55:13.036672 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:13.036527 2549 scope.go:117] "RemoveContainer" containerID="5abaff829723b3fbc09e2043ee6d92792dc5ff5aacf71e9d256ccbdbf3622b59" Jul 22 09:55:13.047933 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:13.047917 2549 scope.go:117] "RemoveContainer" containerID="5abaff829723b3fbc09e2043ee6d92792dc5ff5aacf71e9d256ccbdbf3622b59" Jul 22 09:55:13.048287 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:55:13.048256 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5abaff829723b3fbc09e2043ee6d92792dc5ff5aacf71e9d256ccbdbf3622b59\": container with ID starting with 5abaff829723b3fbc09e2043ee6d92792dc5ff5aacf71e9d256ccbdbf3622b59 not found: ID does not exist" containerID="5abaff829723b3fbc09e2043ee6d92792dc5ff5aacf71e9d256ccbdbf3622b59" Jul 22 09:55:13.048402 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:13.048295 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5abaff829723b3fbc09e2043ee6d92792dc5ff5aacf71e9d256ccbdbf3622b59"} err="failed to get container status \"5abaff829723b3fbc09e2043ee6d92792dc5ff5aacf71e9d256ccbdbf3622b59\": rpc error: code = NotFound desc = could not find container \"5abaff829723b3fbc09e2043ee6d92792dc5ff5aacf71e9d256ccbdbf3622b59\": container with ID starting with 5abaff829723b3fbc09e2043ee6d92792dc5ff5aacf71e9d256ccbdbf3622b59 not found: ID does not exist" Jul 22 09:55:13.060485 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:13.060461 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g"] Jul 22 09:55:13.064007 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:13.063987 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-982fc1-54c7b5655b-6hq4g"] Jul 22 09:55:13.088220 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:13.088200 2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e22d96-c8c6-4567-842a-5b511fedb361-openshift-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:13.088220 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:13.088219 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7xwj\" (UniqueName: \"kubernetes.io/projected/b7e22d96-c8c6-4567-842a-5b511fedb361-kube-api-access-z7xwj\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:13.088344 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:13.088230 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7e22d96-c8c6-4567-842a-5b511fedb361-maas-api-tls\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:14.339240 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:14.339206 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7e22d96-c8c6-4567-842a-5b511fedb361" path="/var/lib/kubelet/pods/b7e22d96-c8c6-4567-842a-5b511fedb361/volumes" Jul 22 09:55:17.185064 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:17.185023 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:17.652957 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:17.652909 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:23.480582 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:23.480551 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:23.582022 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:23.581981 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:23.734554 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:23.734472 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx"] Jul 22 09:55:23.734759 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:23.734733 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" podUID="8e2ec773-7540-4012-8293-ec4fd24a7606" containerName="istio-proxy" containerID="cri-o://2f67c69abaddab4614b76104f5e8ac03d4004f41a91e9d3f22e2349372715745" gracePeriod=30 Jul 22 09:55:23.754741 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:23.754714 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:23.990349 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:23.990244 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:24.742994 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:24.742950 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:24.844196 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:24.844161 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:28.995248 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:28.995228 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:55:29.030180 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.030160 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-credential-socket\") pod \"8e2ec773-7540-4012-8293-ec4fd24a7606\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " Jul 22 09:55:29.030315 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.030198 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-data\") pod \"8e2ec773-7540-4012-8293-ec4fd24a7606\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " Jul 22 09:55:29.030315 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.030223 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-token\") pod \"8e2ec773-7540-4012-8293-ec4fd24a7606\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " Jul 22 09:55:29.030315 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.030269 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-podinfo\") pod \"8e2ec773-7540-4012-8293-ec4fd24a7606\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " Jul 22 09:55:29.030496 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.030346 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2ec773-7540-4012-8293-ec4fd24a7606-istiod-ca-cert\") pod \"8e2ec773-7540-4012-8293-ec4fd24a7606\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " Jul 22 09:55:29.030496 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.030381 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-workload-socket\") pod \"8e2ec773-7540-4012-8293-ec4fd24a7606\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " Jul 22 09:55:29.030496 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.030405 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-envoy\") pod \"8e2ec773-7540-4012-8293-ec4fd24a7606\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " Jul 22 09:55:29.030496 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.030434 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpjvf\" (UniqueName: \"kubernetes.io/projected/8e2ec773-7540-4012-8293-ec4fd24a7606-kube-api-access-wpjvf\") pod \"8e2ec773-7540-4012-8293-ec4fd24a7606\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " Jul 22 09:55:29.030496 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.030424 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8e2ec773-7540-4012-8293-ec4fd24a7606" (UID: "8e2ec773-7540-4012-8293-ec4fd24a7606"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:55:29.030751 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.030503 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-workload-certs\") pod \"8e2ec773-7540-4012-8293-ec4fd24a7606\" (UID: \"8e2ec773-7540-4012-8293-ec4fd24a7606\") " Jul 22 09:55:29.030751 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.030742 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:29.030868 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.030760 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e2ec773-7540-4012-8293-ec4fd24a7606-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8e2ec773-7540-4012-8293-ec4fd24a7606" (UID: "8e2ec773-7540-4012-8293-ec4fd24a7606"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:55:29.031004 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.030954 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8e2ec773-7540-4012-8293-ec4fd24a7606" (UID: "8e2ec773-7540-4012-8293-ec4fd24a7606"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:55:29.031148 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.031112 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8e2ec773-7540-4012-8293-ec4fd24a7606" (UID: "8e2ec773-7540-4012-8293-ec4fd24a7606"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:55:29.031212 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.031159 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-data" (OuterVolumeSpecName: "istio-data") pod "8e2ec773-7540-4012-8293-ec4fd24a7606" (UID: "8e2ec773-7540-4012-8293-ec4fd24a7606"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:55:29.032632 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.032595 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8e2ec773-7540-4012-8293-ec4fd24a7606" (UID: "8e2ec773-7540-4012-8293-ec4fd24a7606"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 09:55:29.032771 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.032750 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2ec773-7540-4012-8293-ec4fd24a7606-kube-api-access-wpjvf" (OuterVolumeSpecName: "kube-api-access-wpjvf") pod "8e2ec773-7540-4012-8293-ec4fd24a7606" (UID: "8e2ec773-7540-4012-8293-ec4fd24a7606"). InnerVolumeSpecName "kube-api-access-wpjvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:55:29.032846 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.032782 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8e2ec773-7540-4012-8293-ec4fd24a7606" (UID: "8e2ec773-7540-4012-8293-ec4fd24a7606"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:55:29.032949 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.032925 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-token" (OuterVolumeSpecName: "istio-token") pod "8e2ec773-7540-4012-8293-ec4fd24a7606" (UID: "8e2ec773-7540-4012-8293-ec4fd24a7606"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:55:29.119710 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.119674 2549 generic.go:356] "Generic (PLEG): container finished" podID="8e2ec773-7540-4012-8293-ec4fd24a7606" containerID="2f67c69abaddab4614b76104f5e8ac03d4004f41a91e9d3f22e2349372715745" exitCode=0 Jul 22 09:55:29.119868 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.119718 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" event={"ID":"8e2ec773-7540-4012-8293-ec4fd24a7606","Type":"ContainerDied","Data":"2f67c69abaddab4614b76104f5e8ac03d4004f41a91e9d3f22e2349372715745"} Jul 22 09:55:29.119868 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.119727 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" Jul 22 09:55:29.119868 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.119765 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx" event={"ID":"8e2ec773-7540-4012-8293-ec4fd24a7606","Type":"ContainerDied","Data":"023a214f801043b4ad39cd7d8c5fd21252e2e9d21f9e4c5ef8d8e637ccc5bb08"} Jul 22 09:55:29.119868 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.119792 2549 scope.go:117] "RemoveContainer" containerID="2f67c69abaddab4614b76104f5e8ac03d4004f41a91e9d3f22e2349372715745" Jul 22 09:55:29.129405 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.129384 2549 scope.go:117] "RemoveContainer" containerID="9f35d9a4a91fcccae8e9b38cbb71f4b55843a86119b6d7433b57f0d23a3d1556" Jul 22 09:55:29.131537 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.131515 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2ec773-7540-4012-8293-ec4fd24a7606-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:29.131537 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.131537 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:29.131665 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.131547 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:29.131665 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.131556 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpjvf\" (UniqueName: \"kubernetes.io/projected/8e2ec773-7540-4012-8293-ec4fd24a7606-kube-api-access-wpjvf\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:29.131665 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.131565 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:29.131665 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.131573 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:29.131665 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.131580 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:29.131665 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.131587 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2ec773-7540-4012-8293-ec4fd24a7606-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:29.136911 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.136893 2549 scope.go:117] "RemoveContainer" containerID="2f67c69abaddab4614b76104f5e8ac03d4004f41a91e9d3f22e2349372715745" Jul 22 09:55:29.137174 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:55:29.137152 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f67c69abaddab4614b76104f5e8ac03d4004f41a91e9d3f22e2349372715745\": container with ID starting with 2f67c69abaddab4614b76104f5e8ac03d4004f41a91e9d3f22e2349372715745 not found: ID does not exist" containerID="2f67c69abaddab4614b76104f5e8ac03d4004f41a91e9d3f22e2349372715745" Jul 22 09:55:29.137226 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.137182 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f67c69abaddab4614b76104f5e8ac03d4004f41a91e9d3f22e2349372715745"} err="failed to get container status \"2f67c69abaddab4614b76104f5e8ac03d4004f41a91e9d3f22e2349372715745\": rpc error: code = NotFound desc = could not find container \"2f67c69abaddab4614b76104f5e8ac03d4004f41a91e9d3f22e2349372715745\": container with ID starting with 2f67c69abaddab4614b76104f5e8ac03d4004f41a91e9d3f22e2349372715745 not found: ID does not exist" Jul 22 09:55:29.137226 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.137200 2549 scope.go:117] "RemoveContainer" containerID="9f35d9a4a91fcccae8e9b38cbb71f4b55843a86119b6d7433b57f0d23a3d1556" Jul 22 09:55:29.137465 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:55:29.137448 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f35d9a4a91fcccae8e9b38cbb71f4b55843a86119b6d7433b57f0d23a3d1556\": container with ID starting with 9f35d9a4a91fcccae8e9b38cbb71f4b55843a86119b6d7433b57f0d23a3d1556 not found: ID does not exist" containerID="9f35d9a4a91fcccae8e9b38cbb71f4b55843a86119b6d7433b57f0d23a3d1556" Jul 22 09:55:29.137513 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.137471 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f35d9a4a91fcccae8e9b38cbb71f4b55843a86119b6d7433b57f0d23a3d1556"} err="failed to get container status \"9f35d9a4a91fcccae8e9b38cbb71f4b55843a86119b6d7433b57f0d23a3d1556\": rpc error: code = NotFound desc = could not find container \"9f35d9a4a91fcccae8e9b38cbb71f4b55843a86119b6d7433b57f0d23a3d1556\": container with ID starting with 9f35d9a4a91fcccae8e9b38cbb71f4b55843a86119b6d7433b57f0d23a3d1556 not found: ID does not exist" Jul 22 09:55:29.143315 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.143281 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx"] Jul 22 09:55:29.148725 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:29.148703 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-982fc1-openshift-default-774b6fd5ff-5vhhx"] Jul 22 09:55:30.339078 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:30.339047 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e2ec773-7540-4012-8293-ec4fd24a7606" path="/var/lib/kubelet/pods/8e2ec773-7540-4012-8293-ec4fd24a7606/volumes" Jul 22 09:55:34.700801 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:34.700763 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb"] Jul 22 09:55:34.701198 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:34.701102 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" podUID="2d6a8929-b466-46ed-8b70-c0063524738d" containerName="maas-api" containerID="cri-o://405a5bdd9d7892e772c955008481302038f7e1313e55c3ec861720a6510d68f1" gracePeriod=30 Jul 22 09:55:34.752776 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:34.752746 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:34.953427 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:34.953363 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" Jul 22 09:55:34.975853 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:34.975824 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d6a8929-b466-46ed-8b70-c0063524738d-maas-api-tls\") pod \"2d6a8929-b466-46ed-8b70-c0063524738d\" (UID: \"2d6a8929-b466-46ed-8b70-c0063524738d\") " Jul 22 09:55:34.976018 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:34.975896 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d6a8929-b466-46ed-8b70-c0063524738d-openshift-service-ca\") pod \"2d6a8929-b466-46ed-8b70-c0063524738d\" (UID: \"2d6a8929-b466-46ed-8b70-c0063524738d\") " Jul 22 09:55:34.976018 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:34.975926 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8l646\" (UniqueName: \"kubernetes.io/projected/2d6a8929-b466-46ed-8b70-c0063524738d-kube-api-access-8l646\") pod \"2d6a8929-b466-46ed-8b70-c0063524738d\" (UID: \"2d6a8929-b466-46ed-8b70-c0063524738d\") " Jul 22 09:55:34.976878 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:34.976847 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d6a8929-b466-46ed-8b70-c0063524738d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2d6a8929-b466-46ed-8b70-c0063524738d" (UID: "2d6a8929-b466-46ed-8b70-c0063524738d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:55:34.978644 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:34.978619 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d6a8929-b466-46ed-8b70-c0063524738d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2d6a8929-b466-46ed-8b70-c0063524738d" (UID: "2d6a8929-b466-46ed-8b70-c0063524738d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:55:34.978750 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:34.978703 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6a8929-b466-46ed-8b70-c0063524738d-kube-api-access-8l646" (OuterVolumeSpecName: "kube-api-access-8l646") pod "2d6a8929-b466-46ed-8b70-c0063524738d" (UID: "2d6a8929-b466-46ed-8b70-c0063524738d"). InnerVolumeSpecName "kube-api-access-8l646". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:55:35.076989 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:35.076950 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d6a8929-b466-46ed-8b70-c0063524738d-maas-api-tls\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:35.076989 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:35.076982 2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d6a8929-b466-46ed-8b70-c0063524738d-openshift-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:35.076989 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:35.076995 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l646\" (UniqueName: \"kubernetes.io/projected/2d6a8929-b466-46ed-8b70-c0063524738d-kube-api-access-8l646\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:35.156102 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:35.156060 2549 generic.go:356] "Generic (PLEG): container finished" podID="2d6a8929-b466-46ed-8b70-c0063524738d" containerID="405a5bdd9d7892e772c955008481302038f7e1313e55c3ec861720a6510d68f1" exitCode=0 Jul 22 09:55:35.156288 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:35.156109 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" Jul 22 09:55:35.156288 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:35.156120 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" event={"ID":"2d6a8929-b466-46ed-8b70-c0063524738d","Type":"ContainerDied","Data":"405a5bdd9d7892e772c955008481302038f7e1313e55c3ec861720a6510d68f1"} Jul 22 09:55:35.156288 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:35.156174 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb" event={"ID":"2d6a8929-b466-46ed-8b70-c0063524738d","Type":"ContainerDied","Data":"319e17ff61ba1fb9dac3db7644ac20df0ea27a26861c004776e1029ca5ec6321"} Jul 22 09:55:35.156288 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:35.156200 2549 scope.go:117] "RemoveContainer" containerID="405a5bdd9d7892e772c955008481302038f7e1313e55c3ec861720a6510d68f1" Jul 22 09:55:35.165531 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:35.165513 2549 scope.go:117] "RemoveContainer" containerID="405a5bdd9d7892e772c955008481302038f7e1313e55c3ec861720a6510d68f1" Jul 22 09:55:35.165807 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:55:35.165786 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"405a5bdd9d7892e772c955008481302038f7e1313e55c3ec861720a6510d68f1\": container with ID starting with 405a5bdd9d7892e772c955008481302038f7e1313e55c3ec861720a6510d68f1 not found: ID does not exist" containerID="405a5bdd9d7892e772c955008481302038f7e1313e55c3ec861720a6510d68f1" Jul 22 09:55:35.165905 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:35.165813 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"405a5bdd9d7892e772c955008481302038f7e1313e55c3ec861720a6510d68f1"} err="failed to get container status \"405a5bdd9d7892e772c955008481302038f7e1313e55c3ec861720a6510d68f1\": rpc error: code = NotFound desc = could not find container \"405a5bdd9d7892e772c955008481302038f7e1313e55c3ec861720a6510d68f1\": container with ID starting with 405a5bdd9d7892e772c955008481302038f7e1313e55c3ec861720a6510d68f1 not found: ID does not exist" Jul 22 09:55:35.178440 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:35.178413 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb"] Jul 22 09:55:35.183617 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:35.183584 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b1d31b-7d8fb8fdf7-bktfb"] Jul 22 09:55:36.339459 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:36.339428 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d6a8929-b466-46ed-8b70-c0063524738d" path="/var/lib/kubelet/pods/2d6a8929-b466-46ed-8b70-c0063524738d/volumes" Jul 22 09:55:39.295774 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:39.295735 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:39.718846 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:39.718759 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:45.432089 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:45.432051 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:45.687744 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:45.687656 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6"] Jul 22 09:55:45.687964 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:45.687922 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" podUID="7f518bf2-4686-487b-acd7-9eaa0257ba50" containerName="istio-proxy" containerID="cri-o://0b188fd954f255de96b4872b6f75248353995e1f4e720c9a7c3d7c2895a011c4" gracePeriod=30 Jul 22 09:55:46.618196 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.618153 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6"] Jul 22 09:55:46.618671 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.618612 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e2ec773-7540-4012-8293-ec4fd24a7606" containerName="istio-proxy" Jul 22 09:55:46.618671 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.618631 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e2ec773-7540-4012-8293-ec4fd24a7606" containerName="istio-proxy" Jul 22 09:55:46.618671 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.618640 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7e22d96-c8c6-4567-842a-5b511fedb361" containerName="maas-api" Jul 22 09:55:46.618671 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.618649 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7e22d96-c8c6-4567-842a-5b511fedb361" containerName="maas-api" Jul 22 09:55:46.618671 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.618660 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d6a8929-b466-46ed-8b70-c0063524738d" containerName="maas-api" Jul 22 09:55:46.618671 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.618668 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d6a8929-b466-46ed-8b70-c0063524738d" containerName="maas-api" Jul 22 09:55:46.618974 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.618697 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e2ec773-7540-4012-8293-ec4fd24a7606" containerName="istio-proxy" Jul 22 09:55:46.618974 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.618707 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e2ec773-7540-4012-8293-ec4fd24a7606" containerName="istio-proxy" Jul 22 09:55:46.618974 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.618789 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e2ec773-7540-4012-8293-ec4fd24a7606" containerName="istio-proxy" Jul 22 09:55:46.618974 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.618801 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7e22d96-c8c6-4567-842a-5b511fedb361" containerName="maas-api" Jul 22 09:55:46.618974 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.618813 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d6a8929-b466-46ed-8b70-c0063524738d" containerName="maas-api" Jul 22 09:55:46.622698 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.622675 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.625460 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.625437 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-86258d-openshift-default-dockercfg-glswh\"" Jul 22 09:55:46.637670 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.637646 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6"] Jul 22 09:55:46.681637 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.681607 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-istio-envoy\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.681810 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.681642 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-workload-certs\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.681810 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.681678 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-workload-socket\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.681810 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.681703 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafb95c1-2950-41fc-a94e-3225ca068664-istio-token\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.681810 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.681724 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-credential-socket\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.681810 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.681784 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafb95c1-2950-41fc-a94e-3225ca068664-istiod-ca-cert\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.681810 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.681811 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-istio-data\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.682029 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.681829 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafb95c1-2950-41fc-a94e-3225ca068664-istio-podinfo\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.682029 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.681868 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wkfb\" (UniqueName: \"kubernetes.io/projected/dafb95c1-2950-41fc-a94e-3225ca068664-kube-api-access-5wkfb\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.685633 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.685607 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:46.771946 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.771917 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:46.782931 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.782897 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafb95c1-2950-41fc-a94e-3225ca068664-istio-token\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.782931 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.782932 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-credential-socket\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.783139 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.782967 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafb95c1-2950-41fc-a94e-3225ca068664-istiod-ca-cert\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.783139 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.782994 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-istio-data\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.783139 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.783014 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafb95c1-2950-41fc-a94e-3225ca068664-istio-podinfo\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.783139 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.783044 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wkfb\" (UniqueName: \"kubernetes.io/projected/dafb95c1-2950-41fc-a94e-3225ca068664-kube-api-access-5wkfb\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.783139 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.783071 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-istio-envoy\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.783139 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.783097 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-workload-certs\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.783139 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.783129 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-workload-socket\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.783505 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.783456 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-credential-socket\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.783584 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.783562 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-workload-socket\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.783661 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.783641 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-workload-certs\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.783725 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.783703 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-istio-data\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.783958 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.783936 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafb95c1-2950-41fc-a94e-3225ca068664-istiod-ca-cert\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.785475 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.785452 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-istio-envoy\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.785575 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.785556 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafb95c1-2950-41fc-a94e-3225ca068664-istio-podinfo\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.796007 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.795979 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafb95c1-2950-41fc-a94e-3225ca068664-istio-token\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.799350 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.799325 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wkfb\" (UniqueName: \"kubernetes.io/projected/dafb95c1-2950-41fc-a94e-3225ca068664-kube-api-access-5wkfb\") pod \"e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:46.858184 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.858150 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:46.932403 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:46.932294 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:47.069986 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:47.069950 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6"] Jul 22 09:55:47.072419 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:55:47.072395 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddafb95c1_2950_41fc_a94e_3225ca068664.slice/crio-1b9bbe4200a06885ac8f23acfb971a1ec401b6034a431d51616df44e3a55069a WatchSource:0}: Error finding container 1b9bbe4200a06885ac8f23acfb971a1ec401b6034a431d51616df44e3a55069a: Status 404 returned error can't find the container with id 1b9bbe4200a06885ac8f23acfb971a1ec401b6034a431d51616df44e3a55069a Jul 22 09:55:47.074444 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:47.074410 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:55:47.074543 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:47.074494 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:55:47.074606 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:47.074542 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:55:47.219715 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:47.219620 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" event={"ID":"dafb95c1-2950-41fc-a94e-3225ca068664","Type":"ContainerStarted","Data":"dec9d4c525c6bfd58087917e0fc3d3edaf886130bf1d03fba7c013afa4b0d5b5"} Jul 22 09:55:47.219715 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:47.219680 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" event={"ID":"dafb95c1-2950-41fc-a94e-3225ca068664","Type":"ContainerStarted","Data":"1b9bbe4200a06885ac8f23acfb971a1ec401b6034a431d51616df44e3a55069a"} Jul 22 09:55:47.240348 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:47.240063 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" podStartSLOduration=1.24004412 podStartE2EDuration="1.24004412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:55:47.239714907 +0000 UTC m=+2277.703509348" watchObservedRunningTime="2026-07-22 09:55:47.24004412 +0000 UTC m=+2277.703838553" Jul 22 09:55:47.932392 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:47.932358 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:48.937054 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:48.937025 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:49.216822 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.216741 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:49.230526 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.230499 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:49.231338 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.231321 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:55:49.303548 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.303512 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v"] Jul 22 09:55:49.309266 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.304262 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e2ec773-7540-4012-8293-ec4fd24a7606" containerName="istio-proxy" Jul 22 09:55:49.312786 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.312756 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v" Jul 22 09:55:49.315242 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.315222 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-86258d-serving-cert\"" Jul 22 09:55:49.315350 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.315229 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-876r5\"" Jul 22 09:55:49.319149 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.319129 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v"] Jul 22 09:55:49.411497 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.411455 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7gwk\" (UniqueName: \"kubernetes.io/projected/5fb5319a-57b4-4ad2-abfb-e30f19748d79-kube-api-access-x7gwk\") pod \"maas-api-e2e-api-a-86258d-5c74787c4c-8j68v\" (UID: \"5fb5319a-57b4-4ad2-abfb-e30f19748d79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v" Jul 22 09:55:49.411685 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.411517 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fb5319a-57b4-4ad2-abfb-e30f19748d79-openshift-service-ca\") pod \"maas-api-e2e-api-a-86258d-5c74787c4c-8j68v\" (UID: \"5fb5319a-57b4-4ad2-abfb-e30f19748d79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v" Jul 22 09:55:49.411817 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.411786 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fb5319a-57b4-4ad2-abfb-e30f19748d79-maas-api-tls\") pod \"maas-api-e2e-api-a-86258d-5c74787c4c-8j68v\" (UID: \"5fb5319a-57b4-4ad2-abfb-e30f19748d79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v" Jul 22 09:55:49.513201 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.513112 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fb5319a-57b4-4ad2-abfb-e30f19748d79-maas-api-tls\") pod \"maas-api-e2e-api-a-86258d-5c74787c4c-8j68v\" (UID: \"5fb5319a-57b4-4ad2-abfb-e30f19748d79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v" Jul 22 09:55:49.513201 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.513166 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7gwk\" (UniqueName: \"kubernetes.io/projected/5fb5319a-57b4-4ad2-abfb-e30f19748d79-kube-api-access-x7gwk\") pod \"maas-api-e2e-api-a-86258d-5c74787c4c-8j68v\" (UID: \"5fb5319a-57b4-4ad2-abfb-e30f19748d79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v" Jul 22 09:55:49.513458 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.513287 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fb5319a-57b4-4ad2-abfb-e30f19748d79-openshift-service-ca\") pod \"maas-api-e2e-api-a-86258d-5c74787c4c-8j68v\" (UID: \"5fb5319a-57b4-4ad2-abfb-e30f19748d79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v" Jul 22 09:55:49.514115 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.514090 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fb5319a-57b4-4ad2-abfb-e30f19748d79-openshift-service-ca\") pod \"maas-api-e2e-api-a-86258d-5c74787c4c-8j68v\" (UID: \"5fb5319a-57b4-4ad2-abfb-e30f19748d79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v" Jul 22 09:55:49.515848 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.515826 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fb5319a-57b4-4ad2-abfb-e30f19748d79-maas-api-tls\") pod \"maas-api-e2e-api-a-86258d-5c74787c4c-8j68v\" (UID: \"5fb5319a-57b4-4ad2-abfb-e30f19748d79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v" Jul 22 09:55:49.522727 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.522705 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7gwk\" (UniqueName: \"kubernetes.io/projected/5fb5319a-57b4-4ad2-abfb-e30f19748d79-kube-api-access-x7gwk\") pod \"maas-api-e2e-api-a-86258d-5c74787c4c-8j68v\" (UID: \"5fb5319a-57b4-4ad2-abfb-e30f19748d79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v" Jul 22 09:55:49.566561 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.566530 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:49.624958 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.624924 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v" Jul 22 09:55:49.663914 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.663863 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:49.751759 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.751737 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v"] Jul 22 09:55:49.753887 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:55:49.753862 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb5319a_57b4_4ad2_abfb_e30f19748d79.slice/crio-97c9bd9eca35512d4dc045cd03c5140af38ad9d1666799860741712a0084e4fa WatchSource:0}: Error finding container 97c9bd9eca35512d4dc045cd03c5140af38ad9d1666799860741712a0084e4fa: Status 404 returned error can't find the container with id 97c9bd9eca35512d4dc045cd03c5140af38ad9d1666799860741712a0084e4fa Jul 22 09:55:49.755661 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:49.755644 2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 09:55:50.016724 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:50.016687 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:50.120802 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:50.120771 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:50.236384 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:50.236336 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v" event={"ID":"5fb5319a-57b4-4ad2-abfb-e30f19748d79","Type":"ContainerStarted","Data":"97c9bd9eca35512d4dc045cd03c5140af38ad9d1666799860741712a0084e4fa"} Jul 22 09:55:50.931174 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:50.931151 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:55:51.026540 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.026508 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-workload-socket\") pod \"7f518bf2-4686-487b-acd7-9eaa0257ba50\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " Jul 22 09:55:51.026967 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.026566 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-data\") pod \"7f518bf2-4686-487b-acd7-9eaa0257ba50\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " Jul 22 09:55:51.026967 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.026591 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-envoy\") pod \"7f518bf2-4686-487b-acd7-9eaa0257ba50\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " Jul 22 09:55:51.026967 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.026621 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-workload-certs\") pod \"7f518bf2-4686-487b-acd7-9eaa0257ba50\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " Jul 22 09:55:51.026967 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.026671 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb9jn\" (UniqueName: \"kubernetes.io/projected/7f518bf2-4686-487b-acd7-9eaa0257ba50-kube-api-access-rb9jn\") pod \"7f518bf2-4686-487b-acd7-9eaa0257ba50\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " Jul 22 09:55:51.026967 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.026697 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-podinfo\") pod \"7f518bf2-4686-487b-acd7-9eaa0257ba50\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " Jul 22 09:55:51.026967 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.026739 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-credential-socket\") pod \"7f518bf2-4686-487b-acd7-9eaa0257ba50\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " Jul 22 09:55:51.026967 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.026826 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7f518bf2-4686-487b-acd7-9eaa0257ba50" (UID: "7f518bf2-4686-487b-acd7-9eaa0257ba50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:55:51.026967 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.026835 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-token\") pod \"7f518bf2-4686-487b-acd7-9eaa0257ba50\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " Jul 22 09:55:51.026967 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.026925 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f518bf2-4686-487b-acd7-9eaa0257ba50-istiod-ca-cert\") pod \"7f518bf2-4686-487b-acd7-9eaa0257ba50\" (UID: \"7f518bf2-4686-487b-acd7-9eaa0257ba50\") " Jul 22 09:55:51.027552 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.027253 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:51.027552 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.027332 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7f518bf2-4686-487b-acd7-9eaa0257ba50" (UID: "7f518bf2-4686-487b-acd7-9eaa0257ba50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:55:51.027552 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.027365 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f518bf2-4686-487b-acd7-9eaa0257ba50-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7f518bf2-4686-487b-acd7-9eaa0257ba50" (UID: "7f518bf2-4686-487b-acd7-9eaa0257ba50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:55:51.027552 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.027433 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7f518bf2-4686-487b-acd7-9eaa0257ba50" (UID: "7f518bf2-4686-487b-acd7-9eaa0257ba50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:55:51.027823 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.027799 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-data" (OuterVolumeSpecName: "istio-data") pod "7f518bf2-4686-487b-acd7-9eaa0257ba50" (UID: "7f518bf2-4686-487b-acd7-9eaa0257ba50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:55:51.028984 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.028961 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-token" (OuterVolumeSpecName: "istio-token") pod "7f518bf2-4686-487b-acd7-9eaa0257ba50" (UID: "7f518bf2-4686-487b-acd7-9eaa0257ba50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:55:51.029131 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.029112 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f518bf2-4686-487b-acd7-9eaa0257ba50-kube-api-access-rb9jn" (OuterVolumeSpecName: "kube-api-access-rb9jn") pod "7f518bf2-4686-487b-acd7-9eaa0257ba50" (UID: "7f518bf2-4686-487b-acd7-9eaa0257ba50"). InnerVolumeSpecName "kube-api-access-rb9jn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:55:51.029232 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.029214 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7f518bf2-4686-487b-acd7-9eaa0257ba50" (UID: "7f518bf2-4686-487b-acd7-9eaa0257ba50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:55:51.029346 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.029327 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7f518bf2-4686-487b-acd7-9eaa0257ba50" (UID: "7f518bf2-4686-487b-acd7-9eaa0257ba50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 09:55:51.127986 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.127907 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:51.127986 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.127934 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:51.127986 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.127946 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb9jn\" (UniqueName: \"kubernetes.io/projected/7f518bf2-4686-487b-acd7-9eaa0257ba50-kube-api-access-rb9jn\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:51.127986 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.127962 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:51.127986 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.127974 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:51.127986 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.127986 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:51.128274 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.127998 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f518bf2-4686-487b-acd7-9eaa0257ba50-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:51.128274 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.128009 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f518bf2-4686-487b-acd7-9eaa0257ba50-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:55:51.243743 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.243713 2549 generic.go:356] "Generic (PLEG): container finished" podID="7f518bf2-4686-487b-acd7-9eaa0257ba50" containerID="0b188fd954f255de96b4872b6f75248353995e1f4e720c9a7c3d7c2895a011c4" exitCode=0 Jul 22 09:55:51.243927 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.243762 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" Jul 22 09:55:51.243927 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.243798 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" event={"ID":"7f518bf2-4686-487b-acd7-9eaa0257ba50","Type":"ContainerDied","Data":"0b188fd954f255de96b4872b6f75248353995e1f4e720c9a7c3d7c2895a011c4"} Jul 22 09:55:51.243927 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.243863 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6" event={"ID":"7f518bf2-4686-487b-acd7-9eaa0257ba50","Type":"ContainerDied","Data":"56f12077cd8dea5bc7be2085dc2c95eba863e9a065bf9f2d9793a6bb1db71dd5"} Jul 22 09:55:51.243927 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.243893 2549 scope.go:117] "RemoveContainer" containerID="0b188fd954f255de96b4872b6f75248353995e1f4e720c9a7c3d7c2895a011c4" Jul 22 09:55:51.253429 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.253415 2549 scope.go:117] "RemoveContainer" containerID="8ed8fd404de3865721b5efcd97101b9d5e747effbef7ad67e1bb244a041fd608" Jul 22 09:55:51.261460 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.261440 2549 scope.go:117] "RemoveContainer" containerID="0b188fd954f255de96b4872b6f75248353995e1f4e720c9a7c3d7c2895a011c4" Jul 22 09:55:51.261701 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:55:51.261683 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b188fd954f255de96b4872b6f75248353995e1f4e720c9a7c3d7c2895a011c4\": container with ID starting with 0b188fd954f255de96b4872b6f75248353995e1f4e720c9a7c3d7c2895a011c4 not found: ID does not exist" containerID="0b188fd954f255de96b4872b6f75248353995e1f4e720c9a7c3d7c2895a011c4" Jul 22 09:55:51.261751 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.261709 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b188fd954f255de96b4872b6f75248353995e1f4e720c9a7c3d7c2895a011c4"} err="failed to get container status \"0b188fd954f255de96b4872b6f75248353995e1f4e720c9a7c3d7c2895a011c4\": rpc error: code = NotFound desc = could not find container \"0b188fd954f255de96b4872b6f75248353995e1f4e720c9a7c3d7c2895a011c4\": container with ID starting with 0b188fd954f255de96b4872b6f75248353995e1f4e720c9a7c3d7c2895a011c4 not found: ID does not exist" Jul 22 09:55:51.261751 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.261726 2549 scope.go:117] "RemoveContainer" containerID="8ed8fd404de3865721b5efcd97101b9d5e747effbef7ad67e1bb244a041fd608" Jul 22 09:55:51.261962 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:55:51.261942 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ed8fd404de3865721b5efcd97101b9d5e747effbef7ad67e1bb244a041fd608\": container with ID starting with 8ed8fd404de3865721b5efcd97101b9d5e747effbef7ad67e1bb244a041fd608 not found: ID does not exist" containerID="8ed8fd404de3865721b5efcd97101b9d5e747effbef7ad67e1bb244a041fd608" Jul 22 09:55:51.262018 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.261972 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ed8fd404de3865721b5efcd97101b9d5e747effbef7ad67e1bb244a041fd608"} err="failed to get container status \"8ed8fd404de3865721b5efcd97101b9d5e747effbef7ad67e1bb244a041fd608\": rpc error: code = NotFound desc = could not find container \"8ed8fd404de3865721b5efcd97101b9d5e747effbef7ad67e1bb244a041fd608\": container with ID starting with 8ed8fd404de3865721b5efcd97101b9d5e747effbef7ad67e1bb244a041fd608 not found: ID does not exist" Jul 22 09:55:51.266322 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.266286 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6"] Jul 22 09:55:51.270140 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:51.270119 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b1d31b-openshift-default-577b88f585-rgkl6"] Jul 22 09:55:52.250558 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:52.250522 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v" event={"ID":"5fb5319a-57b4-4ad2-abfb-e30f19748d79","Type":"ContainerStarted","Data":"84ecf8d61aad44379411cf4616785220a00dd4aab31ad881c6de21afe20bd64e"} Jul 22 09:55:52.250992 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:52.250586 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v" Jul 22 09:55:52.271224 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:52.271174 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v" podStartSLOduration=1.71426805 podStartE2EDuration="3.2711558s" totalImagesPullingTime="1.55688775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:55:52.270140226 +0000 UTC m=+2282.733934688" watchObservedRunningTime="2026-07-22 09:55:52.2711558 +0000 UTC m=+2282.734950239" Jul 22 09:55:52.339652 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:52.339621 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f518bf2-4686-487b-acd7-9eaa0257ba50" path="/var/lib/kubelet/pods/7f518bf2-4686-487b-acd7-9eaa0257ba50/volumes" Jul 22 09:55:53.881825 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:53.881783 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:54.238035 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.237930 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:54.541419 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.541388 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz"] Jul 22 09:55:54.541712 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.541701 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f518bf2-4686-487b-acd7-9eaa0257ba50" containerName="istio-proxy" Jul 22 09:55:54.541765 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.541714 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f518bf2-4686-487b-acd7-9eaa0257ba50" containerName="istio-proxy" Jul 22 09:55:54.541765 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.541724 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f518bf2-4686-487b-acd7-9eaa0257ba50" containerName="istio-proxy" Jul 22 09:55:54.541765 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.541730 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f518bf2-4686-487b-acd7-9eaa0257ba50" containerName="istio-proxy" Jul 22 09:55:54.541866 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.541787 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f518bf2-4686-487b-acd7-9eaa0257ba50" containerName="istio-proxy" Jul 22 09:55:54.541866 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.541796 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f518bf2-4686-487b-acd7-9eaa0257ba50" containerName="istio-proxy" Jul 22 09:55:54.544900 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.544884 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.547561 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.547539 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a5acdb-openshift-default-dockercfg-pqdfr\"" Jul 22 09:55:54.555227 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.555202 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz"] Jul 22 09:55:54.567665 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.567639 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:54.655648 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.655613 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-token\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.655832 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.655664 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-podinfo\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.655832 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.655700 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istiod-ca-cert\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.655832 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.655818 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-workload-socket\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.656033 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.655865 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhr9h\" (UniqueName: \"kubernetes.io/projected/91c30c08-fb9b-45cf-bdc6-f1115ac66394-kube-api-access-jhr9h\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.656033 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.655959 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-data\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.656033 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.655991 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-credential-socket\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.656033 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.656015 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-envoy\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.656209 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.656080 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-workload-certs\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.675772 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.675739 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:54.757246 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.757212 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-workload-certs\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.757417 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.757250 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-token\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.757417 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.757289 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-podinfo\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.757417 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.757329 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istiod-ca-cert\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.757417 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.757366 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-workload-socket\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.757644 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.757417 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhr9h\" (UniqueName: \"kubernetes.io/projected/91c30c08-fb9b-45cf-bdc6-f1115ac66394-kube-api-access-jhr9h\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.757644 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.757517 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-data\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.757644 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.757549 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-credential-socket\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.757644 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.757570 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-envoy\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.757841 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.757693 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-workload-certs\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.757841 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.757709 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-workload-socket\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.757980 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.757950 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-data\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.758088 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.757958 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-credential-socket\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.758133 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.758094 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istiod-ca-cert\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.759771 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.759753 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-envoy\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.759883 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.759866 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-podinfo\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.765594 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.765572 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-token\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.765677 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.765663 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhr9h\" (UniqueName: \"kubernetes.io/projected/91c30c08-fb9b-45cf-bdc6-f1115ac66394-kube-api-access-jhr9h\") pod \"e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:54.855689 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:54.855612 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:55.011432 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:55.011406 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz"] Jul 22 09:55:55.013991 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:55:55.013965 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c30c08_fb9b_45cf_bdc6_f1115ac66394.slice/crio-ea6d41f38694edfa937c0bb2d9621b76c757a87a595ce64bfae8f2f200e2d42e WatchSource:0}: Error finding container ea6d41f38694edfa937c0bb2d9621b76c757a87a595ce64bfae8f2f200e2d42e: Status 404 returned error can't find the container with id ea6d41f38694edfa937c0bb2d9621b76c757a87a595ce64bfae8f2f200e2d42e Jul 22 09:55:55.019877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:55.019842 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:55:55.019984 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:55.019922 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:55:55.019984 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:55.019965 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:55:55.268011 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:55.267927 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" event={"ID":"91c30c08-fb9b-45cf-bdc6-f1115ac66394","Type":"ContainerStarted","Data":"0317d270f2ab419a7021307754f8bc347433982a6e1b75b9d83c74654a0fe76c"} Jul 22 09:55:55.268011 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:55.267977 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" event={"ID":"91c30c08-fb9b-45cf-bdc6-f1115ac66394","Type":"ContainerStarted","Data":"ea6d41f38694edfa937c0bb2d9621b76c757a87a595ce64bfae8f2f200e2d42e"} Jul 22 09:55:55.304399 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:55.304336 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" podStartSLOduration=1.304299986 podStartE2EDuration="1.304299986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:55:55.302697822 +0000 UTC m=+2285.766492272" watchObservedRunningTime="2026-07-22 09:55:55.304299986 +0000 UTC m=+2285.768111062" Jul 22 09:55:55.856244 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:55.856213 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:56.862134 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:56.862106 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:57.279362 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:57.279333 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:57.280143 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:57.280126 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:55:57.314758 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:57.314725 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:57.449433 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:57.449399 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd"] Jul 22 09:55:57.453217 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:57.453191 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" Jul 22 09:55:57.456950 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:57.456924 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a5acdb-serving-cert\"" Jul 22 09:55:57.465369 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:57.465345 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd"] Jul 22 09:55:57.581277 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:57.581187 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-969hr\" (UniqueName: \"kubernetes.io/projected/f1c9952a-d4ab-4a63-875d-c247a58dea81-kube-api-access-969hr\") pod \"maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd\" (UID: \"f1c9952a-d4ab-4a63-875d-c247a58dea81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" Jul 22 09:55:57.581277 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:57.581232 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1c9952a-d4ab-4a63-875d-c247a58dea81-maas-api-tls\") pod \"maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd\" (UID: \"f1c9952a-d4ab-4a63-875d-c247a58dea81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" Jul 22 09:55:57.581519 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:57.581363 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c9952a-d4ab-4a63-875d-c247a58dea81-openshift-service-ca\") pod \"maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd\" (UID: \"f1c9952a-d4ab-4a63-875d-c247a58dea81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" Jul 22 09:55:57.654797 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:57.654765 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:57.682663 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:57.682630 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c9952a-d4ab-4a63-875d-c247a58dea81-openshift-service-ca\") pod \"maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd\" (UID: \"f1c9952a-d4ab-4a63-875d-c247a58dea81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" Jul 22 09:55:57.682818 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:57.682681 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-969hr\" (UniqueName: \"kubernetes.io/projected/f1c9952a-d4ab-4a63-875d-c247a58dea81-kube-api-access-969hr\") pod \"maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd\" (UID: \"f1c9952a-d4ab-4a63-875d-c247a58dea81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" Jul 22 09:55:57.682818 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:57.682704 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1c9952a-d4ab-4a63-875d-c247a58dea81-maas-api-tls\") pod \"maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd\" (UID: \"f1c9952a-d4ab-4a63-875d-c247a58dea81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" Jul 22 09:55:57.682818 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:55:57.682799 2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-serving-cert: secret "maas-api-e2e-api-b-a5acdb-serving-cert" not found Jul 22 09:55:57.682925 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:55:57.682856 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1c9952a-d4ab-4a63-875d-c247a58dea81-maas-api-tls podName:f1c9952a-d4ab-4a63-875d-c247a58dea81 nodeName:}" failed. No retries permitted until 2026-07-22 09:55:58.182841034 +0000 UTC m=+2288.646635449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f1c9952a-d4ab-4a63-875d-c247a58dea81-maas-api-tls") pod "maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" (UID: "f1c9952a-d4ab-4a63-875d-c247a58dea81") : secret "maas-api-e2e-api-b-a5acdb-serving-cert" not found Jul 22 09:55:57.683250 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:57.683231 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c9952a-d4ab-4a63-875d-c247a58dea81-openshift-service-ca\") pod \"maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd\" (UID: \"f1c9952a-d4ab-4a63-875d-c247a58dea81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" Jul 22 09:55:57.706834 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:57.706798 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-969hr\" (UniqueName: \"kubernetes.io/projected/f1c9952a-d4ab-4a63-875d-c247a58dea81-kube-api-access-969hr\") pod \"maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd\" (UID: \"f1c9952a-d4ab-4a63-875d-c247a58dea81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" Jul 22 09:55:57.754692 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:57.754660 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:58.152318 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:58.152284 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:58.188548 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:58.188519 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1c9952a-d4ab-4a63-875d-c247a58dea81-maas-api-tls\") pod \"maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd\" (UID: \"f1c9952a-d4ab-4a63-875d-c247a58dea81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" Jul 22 09:55:58.190963 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:58.190942 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1c9952a-d4ab-4a63-875d-c247a58dea81-maas-api-tls\") pod \"maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd\" (UID: \"f1c9952a-d4ab-4a63-875d-c247a58dea81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" Jul 22 09:55:58.222218 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:58.222181 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:55:58.261984 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:58.261952 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v" Jul 22 09:55:58.368646 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:58.368611 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" Jul 22 09:55:58.501765 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:58.501739 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd"] Jul 22 09:55:58.503735 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:55:58.503711 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c9952a_d4ab_4a63_875d_c247a58dea81.slice/crio-56cf24ad8114dd40e98e61de2d2b1fb6f30f7093f57ac741e5319c61dc6a5b4f WatchSource:0}: Error finding container 56cf24ad8114dd40e98e61de2d2b1fb6f30f7093f57ac741e5319c61dc6a5b4f: Status 404 returned error can't find the container with id 56cf24ad8114dd40e98e61de2d2b1fb6f30f7093f57ac741e5319c61dc6a5b4f Jul 22 09:55:59.292361 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:55:59.292319 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" event={"ID":"f1c9952a-d4ab-4a63-875d-c247a58dea81","Type":"ContainerStarted","Data":"56cf24ad8114dd40e98e61de2d2b1fb6f30f7093f57ac741e5319c61dc6a5b4f"} Jul 22 09:56:01.303508 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:01.303473 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" event={"ID":"f1c9952a-d4ab-4a63-875d-c247a58dea81","Type":"ContainerStarted","Data":"a7062693b427f3217ec37001777a92a75e3f6258838dc1dbfebf973fa54acae1"} Jul 22 09:56:01.303889 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:01.303541 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" Jul 22 09:56:01.325171 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:01.325125 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" podStartSLOduration=2.104302105 podStartE2EDuration="4.325110281s" totalImagesPullingTime="2.220808176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:55:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:56:01.322459674 +0000 UTC m=+2291.786254122" watchObservedRunningTime="2026-07-22 09:56:01.325110281 +0000 UTC m=+2291.788904719" Jul 22 09:56:01.909204 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:01.909175 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:56:05.272510 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:05.272459 2549 patch_prober.go:28] interesting pod/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 22 09:56:05.272902 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:05.272559 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" podUID="dafb95c1-2950-41fc-a94e-3225ca068664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 22 09:56:06.334848 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:06.334812 2549 generic.go:356] "Generic (PLEG): container finished" podID="dafb95c1-2950-41fc-a94e-3225ca068664" containerID="dec9d4c525c6bfd58087917e0fc3d3edaf886130bf1d03fba7c013afa4b0d5b5" exitCode=0 Jul 22 09:56:06.338511 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:06.338487 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" event={"ID":"dafb95c1-2950-41fc-a94e-3225ca068664","Type":"ContainerDied","Data":"dec9d4c525c6bfd58087917e0fc3d3edaf886130bf1d03fba7c013afa4b0d5b5"} Jul 22 09:56:06.338872 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:06.338859 2549 scope.go:117] "RemoveContainer" containerID="dec9d4c525c6bfd58087917e0fc3d3edaf886130bf1d03fba7c013afa4b0d5b5" Jul 22 09:56:06.339896 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:06.339867 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:56:06.339963 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:06.339948 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:56:06.340002 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:06.339987 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:56:07.314154 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:07.314126 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" Jul 22 09:56:07.342465 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:07.342435 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" event={"ID":"dafb95c1-2950-41fc-a94e-3225ca068664","Type":"ContainerStarted","Data":"12478caabd1f0cca2f80bc81c7b631d1d765ab7d2f4202a6388672f0e1c18cfd"} Jul 22 09:56:07.932679 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:07.932639 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:56:07.932884 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:07.932861 2549 patch_prober.go:28] interesting pod/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 22 09:56:07.932937 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:07.932919 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" podUID="dafb95c1-2950-41fc-a94e-3225ca068664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 22 09:56:08.933523 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:08.933484 2549 patch_prober.go:28] interesting pod/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 22 09:56:08.933918 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:08.933541 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" podUID="dafb95c1-2950-41fc-a94e-3225ca068664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 22 09:56:09.933095 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:09.933055 2549 patch_prober.go:28] interesting pod/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 22 09:56:09.933284 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:09.933111 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" podUID="dafb95c1-2950-41fc-a94e-3225ca068664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 22 09:56:10.578269 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:10.578232 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:56:10.687974 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:10.687928 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:56:10.932897 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:10.932800 2549 patch_prober.go:28] interesting pod/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 22 09:56:10.932897 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:10.932865 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" podUID="dafb95c1-2950-41fc-a94e-3225ca068664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 22 09:56:11.932887 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:11.932852 2549 patch_prober.go:28] interesting pod/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 22 09:56:11.933256 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:11.932905 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" podUID="dafb95c1-2950-41fc-a94e-3225ca068664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 22 09:56:12.933120 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:12.933085 2549 patch_prober.go:28] interesting pod/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 22 09:56:12.933602 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:12.933142 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" podUID="dafb95c1-2950-41fc-a94e-3225ca068664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 22 09:56:13.321556 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:13.321516 2549 patch_prober.go:28] interesting pod/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 22 09:56:13.321747 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:13.321583 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" podUID="91c30c08-fb9b-45cf-bdc6-f1115ac66394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 22 09:56:13.936667 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:13.936631 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:56:13.937129 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:13.937000 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:56:13.937632 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:13.937614 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:56:15.386688 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:15.386594 2549 generic.go:356] "Generic (PLEG): container finished" podID="91c30c08-fb9b-45cf-bdc6-f1115ac66394" containerID="0317d270f2ab419a7021307754f8bc347433982a6e1b75b9d83c74654a0fe76c" exitCode=0 Jul 22 09:56:15.386688 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:15.386669 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" event={"ID":"91c30c08-fb9b-45cf-bdc6-f1115ac66394","Type":"ContainerDied","Data":"0317d270f2ab419a7021307754f8bc347433982a6e1b75b9d83c74654a0fe76c"} Jul 22 09:56:15.387130 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:15.387113 2549 scope.go:117] "RemoveContainer" containerID="0317d270f2ab419a7021307754f8bc347433982a6e1b75b9d83c74654a0fe76c" Jul 22 09:56:15.388046 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:15.388020 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:56:15.388102 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:15.388085 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:56:15.388135 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:15.388115 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:56:16.393775 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:16.393736 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" event={"ID":"91c30c08-fb9b-45cf-bdc6-f1115ac66394","Type":"ContainerStarted","Data":"388244e334da0e90fcca4f3801a0620e0742b1a583d339a90884dc94e35619f2"} Jul 22 09:56:16.856216 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:16.856179 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:56:16.856491 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:16.856461 2549 patch_prober.go:28] interesting pod/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 22 09:56:16.856543 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:16.856513 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" podUID="91c30c08-fb9b-45cf-bdc6-f1115ac66394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 22 09:56:17.856474 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:17.856438 2549 patch_prober.go:28] interesting pod/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 22 09:56:17.856903 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:17.856507 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" podUID="91c30c08-fb9b-45cf-bdc6-f1115ac66394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 22 09:56:18.856077 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:18.856041 2549 patch_prober.go:28] interesting pod/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 22 09:56:18.856258 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:18.856108 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" podUID="91c30c08-fb9b-45cf-bdc6-f1115ac66394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 22 09:56:18.921180 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:18.921141 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:56:19.023197 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:19.023158 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:56:19.856243 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:19.856205 2549 patch_prober.go:28] interesting pod/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 22 09:56:19.856442 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:19.856266 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" podUID="91c30c08-fb9b-45cf-bdc6-f1115ac66394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 22 09:56:20.613277 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:20.613243 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:56:20.736534 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:20.736498 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:56:20.859626 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:20.859596 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:56:20.859927 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:20.859910 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:56:20.860693 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:20.860672 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:56:25.845412 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:25.845376 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v"] Jul 22 09:56:25.845826 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:25.845624 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v" podUID="5fb5319a-57b4-4ad2-abfb-e30f19748d79" containerName="maas-api" containerID="cri-o://84ecf8d61aad44379411cf4616785220a00dd4aab31ad881c6de21afe20bd64e" gracePeriod=30 Jul 22 09:56:25.880764 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:25.880737 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:56:26.099621 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:26.099562 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v" Jul 22 09:56:26.143586 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:26.143557 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fb5319a-57b4-4ad2-abfb-e30f19748d79-openshift-service-ca\") pod \"5fb5319a-57b4-4ad2-abfb-e30f19748d79\" (UID: \"5fb5319a-57b4-4ad2-abfb-e30f19748d79\") " Jul 22 09:56:26.143769 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:26.143606 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7gwk\" (UniqueName: \"kubernetes.io/projected/5fb5319a-57b4-4ad2-abfb-e30f19748d79-kube-api-access-x7gwk\") pod \"5fb5319a-57b4-4ad2-abfb-e30f19748d79\" (UID: \"5fb5319a-57b4-4ad2-abfb-e30f19748d79\") " Jul 22 09:56:26.143769 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:26.143690 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fb5319a-57b4-4ad2-abfb-e30f19748d79-maas-api-tls\") pod \"5fb5319a-57b4-4ad2-abfb-e30f19748d79\" (UID: \"5fb5319a-57b4-4ad2-abfb-e30f19748d79\") " Jul 22 09:56:26.143974 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:26.143945 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fb5319a-57b4-4ad2-abfb-e30f19748d79-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5fb5319a-57b4-4ad2-abfb-e30f19748d79" (UID: "5fb5319a-57b4-4ad2-abfb-e30f19748d79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:56:26.145759 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:26.145732 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb5319a-57b4-4ad2-abfb-e30f19748d79-kube-api-access-x7gwk" (OuterVolumeSpecName: "kube-api-access-x7gwk") pod "5fb5319a-57b4-4ad2-abfb-e30f19748d79" (UID: "5fb5319a-57b4-4ad2-abfb-e30f19748d79"). InnerVolumeSpecName "kube-api-access-x7gwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:56:26.145925 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:26.145895 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fb5319a-57b4-4ad2-abfb-e30f19748d79-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5fb5319a-57b4-4ad2-abfb-e30f19748d79" (UID: "5fb5319a-57b4-4ad2-abfb-e30f19748d79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:56:26.244915 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:26.244883 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fb5319a-57b4-4ad2-abfb-e30f19748d79-maas-api-tls\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:56:26.244915 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:26.244907 2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fb5319a-57b4-4ad2-abfb-e30f19748d79-openshift-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:56:26.244915 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:26.244917 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7gwk\" (UniqueName: \"kubernetes.io/projected/5fb5319a-57b4-4ad2-abfb-e30f19748d79-kube-api-access-x7gwk\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:56:26.446660 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:26.446579 2549 generic.go:356] "Generic (PLEG): container finished" podID="5fb5319a-57b4-4ad2-abfb-e30f19748d79" containerID="84ecf8d61aad44379411cf4616785220a00dd4aab31ad881c6de21afe20bd64e" exitCode=0 Jul 22 09:56:26.446660 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:26.446623 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v" event={"ID":"5fb5319a-57b4-4ad2-abfb-e30f19748d79","Type":"ContainerDied","Data":"84ecf8d61aad44379411cf4616785220a00dd4aab31ad881c6de21afe20bd64e"} Jul 22 09:56:26.446660 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:26.446631 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v" Jul 22 09:56:26.446878 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:26.446674 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v" event={"ID":"5fb5319a-57b4-4ad2-abfb-e30f19748d79","Type":"ContainerDied","Data":"97c9bd9eca35512d4dc045cd03c5140af38ad9d1666799860741712a0084e4fa"} Jul 22 09:56:26.446878 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:26.446695 2549 scope.go:117] "RemoveContainer" containerID="84ecf8d61aad44379411cf4616785220a00dd4aab31ad881c6de21afe20bd64e" Jul 22 09:56:26.457482 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:26.457464 2549 scope.go:117] "RemoveContainer" containerID="84ecf8d61aad44379411cf4616785220a00dd4aab31ad881c6de21afe20bd64e" Jul 22 09:56:26.457765 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:56:26.457748 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ecf8d61aad44379411cf4616785220a00dd4aab31ad881c6de21afe20bd64e\": container with ID starting with 84ecf8d61aad44379411cf4616785220a00dd4aab31ad881c6de21afe20bd64e not found: ID does not exist" containerID="84ecf8d61aad44379411cf4616785220a00dd4aab31ad881c6de21afe20bd64e" Jul 22 09:56:26.457813 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:26.457777 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ecf8d61aad44379411cf4616785220a00dd4aab31ad881c6de21afe20bd64e"} err="failed to get container status \"84ecf8d61aad44379411cf4616785220a00dd4aab31ad881c6de21afe20bd64e\": rpc error: code = NotFound desc = could not find container \"84ecf8d61aad44379411cf4616785220a00dd4aab31ad881c6de21afe20bd64e\": container with ID starting with 84ecf8d61aad44379411cf4616785220a00dd4aab31ad881c6de21afe20bd64e not found: ID does not exist" Jul 22 09:56:26.468757 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:26.468734 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v"] Jul 22 09:56:26.475488 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:26.475468 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86258d-5c74787c4c-8j68v"] Jul 22 09:56:28.339922 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:28.339885 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fb5319a-57b4-4ad2-abfb-e30f19748d79" path="/var/lib/kubelet/pods/5fb5319a-57b4-4ad2-abfb-e30f19748d79/volumes" Jul 22 09:56:30.098848 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:30.098812 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:56:30.497807 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:30.497721 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:56:36.231078 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:36.231041 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:56:36.293716 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:36.293689 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:56:36.486534 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:36.486453 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6"] Jul 22 09:56:36.486708 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:36.486682 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" podUID="dafb95c1-2950-41fc-a94e-3225ca068664" containerName="istio-proxy" containerID="cri-o://12478caabd1f0cca2f80bc81c7b631d1d765ab7d2f4202a6388672f0e1c18cfd" gracePeriod=30 Jul 22 09:56:36.514920 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:36.514886 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:56:36.806982 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:36.806944 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:56:37.520236 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:37.520204 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:56:37.620575 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:37.620544 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:56:37.722225 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:37.722197 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:56:41.741686 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.741662 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:56:41.879155 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.879123 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-credential-socket\") pod \"dafb95c1-2950-41fc-a94e-3225ca068664\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " Jul 22 09:56:41.879359 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.879166 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-workload-socket\") pod \"dafb95c1-2950-41fc-a94e-3225ca068664\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " Jul 22 09:56:41.879359 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.879206 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafb95c1-2950-41fc-a94e-3225ca068664-istiod-ca-cert\") pod \"dafb95c1-2950-41fc-a94e-3225ca068664\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " Jul 22 09:56:41.879359 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.879238 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-istio-data\") pod \"dafb95c1-2950-41fc-a94e-3225ca068664\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " Jul 22 09:56:41.879359 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.879272 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wkfb\" (UniqueName: \"kubernetes.io/projected/dafb95c1-2950-41fc-a94e-3225ca068664-kube-api-access-5wkfb\") pod \"dafb95c1-2950-41fc-a94e-3225ca068664\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " Jul 22 09:56:41.879359 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.879328 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafb95c1-2950-41fc-a94e-3225ca068664-istio-token\") pod \"dafb95c1-2950-41fc-a94e-3225ca068664\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " Jul 22 09:56:41.880002 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.879367 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-istio-envoy\") pod \"dafb95c1-2950-41fc-a94e-3225ca068664\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " Jul 22 09:56:41.880002 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.879396 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafb95c1-2950-41fc-a94e-3225ca068664-istio-podinfo\") pod \"dafb95c1-2950-41fc-a94e-3225ca068664\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " Jul 22 09:56:41.880002 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.879446 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dafb95c1-2950-41fc-a94e-3225ca068664" (UID: "dafb95c1-2950-41fc-a94e-3225ca068664"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:56:41.880002 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.879469 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-workload-certs\") pod \"dafb95c1-2950-41fc-a94e-3225ca068664\" (UID: \"dafb95c1-2950-41fc-a94e-3225ca068664\") " Jul 22 09:56:41.880002 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.879724 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:56:41.880258 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.880201 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dafb95c1-2950-41fc-a94e-3225ca068664" (UID: "dafb95c1-2950-41fc-a94e-3225ca068664"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:56:41.880334 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.880281 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafb95c1-2950-41fc-a94e-3225ca068664-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dafb95c1-2950-41fc-a94e-3225ca068664" (UID: "dafb95c1-2950-41fc-a94e-3225ca068664"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:56:41.880400 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.880367 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dafb95c1-2950-41fc-a94e-3225ca068664" (UID: "dafb95c1-2950-41fc-a94e-3225ca068664"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:56:41.881014 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.880984 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-istio-data" (OuterVolumeSpecName: "istio-data") pod "dafb95c1-2950-41fc-a94e-3225ca068664" (UID: "dafb95c1-2950-41fc-a94e-3225ca068664"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:56:41.881974 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.881942 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafb95c1-2950-41fc-a94e-3225ca068664-kube-api-access-5wkfb" (OuterVolumeSpecName: "kube-api-access-5wkfb") pod "dafb95c1-2950-41fc-a94e-3225ca068664" (UID: "dafb95c1-2950-41fc-a94e-3225ca068664"). InnerVolumeSpecName "kube-api-access-5wkfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:56:41.882231 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.882204 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dafb95c1-2950-41fc-a94e-3225ca068664-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dafb95c1-2950-41fc-a94e-3225ca068664" (UID: "dafb95c1-2950-41fc-a94e-3225ca068664"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 09:56:41.882432 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.882409 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dafb95c1-2950-41fc-a94e-3225ca068664" (UID: "dafb95c1-2950-41fc-a94e-3225ca068664"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:56:41.882534 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.882509 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafb95c1-2950-41fc-a94e-3225ca068664-istio-token" (OuterVolumeSpecName: "istio-token") pod "dafb95c1-2950-41fc-a94e-3225ca068664" (UID: "dafb95c1-2950-41fc-a94e-3225ca068664"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:56:41.980444 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.980395 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:56:41.980444 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.980438 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:56:41.980444 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.980449 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafb95c1-2950-41fc-a94e-3225ca068664-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:56:41.980444 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.980458 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:56:41.980723 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.980466 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wkfb\" (UniqueName: \"kubernetes.io/projected/dafb95c1-2950-41fc-a94e-3225ca068664-kube-api-access-5wkfb\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:56:41.980723 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.980476 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafb95c1-2950-41fc-a94e-3225ca068664-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:56:41.980723 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.980483 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafb95c1-2950-41fc-a94e-3225ca068664-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:56:41.980723 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:41.980491 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafb95c1-2950-41fc-a94e-3225ca068664-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:56:42.533898 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:42.533866 2549 generic.go:356] "Generic (PLEG): container finished" podID="dafb95c1-2950-41fc-a94e-3225ca068664" containerID="12478caabd1f0cca2f80bc81c7b631d1d765ab7d2f4202a6388672f0e1c18cfd" exitCode=0 Jul 22 09:56:42.534068 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:42.533912 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" Jul 22 09:56:42.534068 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:42.533913 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" event={"ID":"dafb95c1-2950-41fc-a94e-3225ca068664","Type":"ContainerDied","Data":"12478caabd1f0cca2f80bc81c7b631d1d765ab7d2f4202a6388672f0e1c18cfd"} Jul 22 09:56:42.534068 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:42.534043 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6" event={"ID":"dafb95c1-2950-41fc-a94e-3225ca068664","Type":"ContainerDied","Data":"1b9bbe4200a06885ac8f23acfb971a1ec401b6034a431d51616df44e3a55069a"} Jul 22 09:56:42.534175 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:42.534072 2549 scope.go:117] "RemoveContainer" containerID="12478caabd1f0cca2f80bc81c7b631d1d765ab7d2f4202a6388672f0e1c18cfd" Jul 22 09:56:42.543021 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:42.543006 2549 scope.go:117] "RemoveContainer" containerID="dec9d4c525c6bfd58087917e0fc3d3edaf886130bf1d03fba7c013afa4b0d5b5" Jul 22 09:56:42.550879 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:42.550861 2549 scope.go:117] "RemoveContainer" containerID="12478caabd1f0cca2f80bc81c7b631d1d765ab7d2f4202a6388672f0e1c18cfd" Jul 22 09:56:42.551133 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:56:42.551116 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12478caabd1f0cca2f80bc81c7b631d1d765ab7d2f4202a6388672f0e1c18cfd\": container with ID starting with 12478caabd1f0cca2f80bc81c7b631d1d765ab7d2f4202a6388672f0e1c18cfd not found: ID does not exist" containerID="12478caabd1f0cca2f80bc81c7b631d1d765ab7d2f4202a6388672f0e1c18cfd" Jul 22 09:56:42.551178 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:42.551139 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12478caabd1f0cca2f80bc81c7b631d1d765ab7d2f4202a6388672f0e1c18cfd"} err="failed to get container status \"12478caabd1f0cca2f80bc81c7b631d1d765ab7d2f4202a6388672f0e1c18cfd\": rpc error: code = NotFound desc = could not find container \"12478caabd1f0cca2f80bc81c7b631d1d765ab7d2f4202a6388672f0e1c18cfd\": container with ID starting with 12478caabd1f0cca2f80bc81c7b631d1d765ab7d2f4202a6388672f0e1c18cfd not found: ID does not exist" Jul 22 09:56:42.551178 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:42.551156 2549 scope.go:117] "RemoveContainer" containerID="dec9d4c525c6bfd58087917e0fc3d3edaf886130bf1d03fba7c013afa4b0d5b5" Jul 22 09:56:42.551374 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:56:42.551356 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec9d4c525c6bfd58087917e0fc3d3edaf886130bf1d03fba7c013afa4b0d5b5\": container with ID starting with dec9d4c525c6bfd58087917e0fc3d3edaf886130bf1d03fba7c013afa4b0d5b5 not found: ID does not exist" containerID="dec9d4c525c6bfd58087917e0fc3d3edaf886130bf1d03fba7c013afa4b0d5b5" Jul 22 09:56:42.551424 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:42.551383 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec9d4c525c6bfd58087917e0fc3d3edaf886130bf1d03fba7c013afa4b0d5b5"} err="failed to get container status \"dec9d4c525c6bfd58087917e0fc3d3edaf886130bf1d03fba7c013afa4b0d5b5\": rpc error: code = NotFound desc = could not find container \"dec9d4c525c6bfd58087917e0fc3d3edaf886130bf1d03fba7c013afa4b0d5b5\": container with ID starting with dec9d4c525c6bfd58087917e0fc3d3edaf886130bf1d03fba7c013afa4b0d5b5 not found: ID does not exist" Jul 22 09:56:42.557510 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:42.557485 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6"] Jul 22 09:56:42.560505 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:42.560485 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-86258d-openshift-default-6948bbcf77-4h8l6"] Jul 22 09:56:44.339803 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:44.339768 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dafb95c1-2950-41fc-a94e-3225ca068664" path="/var/lib/kubelet/pods/dafb95c1-2950-41fc-a94e-3225ca068664/volumes" Jul 22 09:56:52.472819 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:52.472784 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd"] Jul 22 09:56:52.473247 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:52.473001 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" podUID="f1c9952a-d4ab-4a63-875d-c247a58dea81" containerName="maas-api" containerID="cri-o://a7062693b427f3217ec37001777a92a75e3f6258838dc1dbfebf973fa54acae1" gracePeriod=30 Jul 22 09:56:52.502192 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:52.502161 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:56:52.590172 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:52.590146 2549 generic.go:356] "Generic (PLEG): container finished" podID="f1c9952a-d4ab-4a63-875d-c247a58dea81" containerID="a7062693b427f3217ec37001777a92a75e3f6258838dc1dbfebf973fa54acae1" exitCode=0 Jul 22 09:56:52.590285 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:52.590218 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" event={"ID":"f1c9952a-d4ab-4a63-875d-c247a58dea81","Type":"ContainerDied","Data":"a7062693b427f3217ec37001777a92a75e3f6258838dc1dbfebf973fa54acae1"} Jul 22 09:56:52.719667 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:52.719644 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" Jul 22 09:56:52.873796 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:52.873751 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c9952a-d4ab-4a63-875d-c247a58dea81-openshift-service-ca\") pod \"f1c9952a-d4ab-4a63-875d-c247a58dea81\" (UID: \"f1c9952a-d4ab-4a63-875d-c247a58dea81\") " Jul 22 09:56:52.874008 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:52.873830 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-969hr\" (UniqueName: \"kubernetes.io/projected/f1c9952a-d4ab-4a63-875d-c247a58dea81-kube-api-access-969hr\") pod \"f1c9952a-d4ab-4a63-875d-c247a58dea81\" (UID: \"f1c9952a-d4ab-4a63-875d-c247a58dea81\") " Jul 22 09:56:52.874008 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:52.873908 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1c9952a-d4ab-4a63-875d-c247a58dea81-maas-api-tls\") pod \"f1c9952a-d4ab-4a63-875d-c247a58dea81\" (UID: \"f1c9952a-d4ab-4a63-875d-c247a58dea81\") " Jul 22 09:56:52.874197 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:52.874173 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c9952a-d4ab-4a63-875d-c247a58dea81-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f1c9952a-d4ab-4a63-875d-c247a58dea81" (UID: "f1c9952a-d4ab-4a63-875d-c247a58dea81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:56:52.875915 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:52.875884 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1c9952a-d4ab-4a63-875d-c247a58dea81-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f1c9952a-d4ab-4a63-875d-c247a58dea81" (UID: "f1c9952a-d4ab-4a63-875d-c247a58dea81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:56:52.876034 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:52.876017 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c9952a-d4ab-4a63-875d-c247a58dea81-kube-api-access-969hr" (OuterVolumeSpecName: "kube-api-access-969hr") pod "f1c9952a-d4ab-4a63-875d-c247a58dea81" (UID: "f1c9952a-d4ab-4a63-875d-c247a58dea81"). InnerVolumeSpecName "kube-api-access-969hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:56:52.975573 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:52.975529 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1c9952a-d4ab-4a63-875d-c247a58dea81-maas-api-tls\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:56:52.975573 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:52.975570 2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c9952a-d4ab-4a63-875d-c247a58dea81-openshift-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:56:52.975573 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:52.975581 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-969hr\" (UniqueName: \"kubernetes.io/projected/f1c9952a-d4ab-4a63-875d-c247a58dea81-kube-api-access-969hr\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:56:53.596694 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:53.596666 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" Jul 22 09:56:53.597103 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:53.596703 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd" event={"ID":"f1c9952a-d4ab-4a63-875d-c247a58dea81","Type":"ContainerDied","Data":"56cf24ad8114dd40e98e61de2d2b1fb6f30f7093f57ac741e5319c61dc6a5b4f"} Jul 22 09:56:53.597103 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:53.596766 2549 scope.go:117] "RemoveContainer" containerID="a7062693b427f3217ec37001777a92a75e3f6258838dc1dbfebf973fa54acae1" Jul 22 09:56:53.618145 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:53.618120 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd"] Jul 22 09:56:53.621601 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:53.621576 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a5acdb-7d478bffc5-kdjtd"] Jul 22 09:56:54.339357 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:54.339298 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1c9952a-d4ab-4a63-875d-c247a58dea81" path="/var/lib/kubelet/pods/f1c9952a-d4ab-4a63-875d-c247a58dea81/volumes" Jul 22 09:56:57.163200 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:57.163163 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:56:57.452069 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:56:57.451984 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:03.645882 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:03.645846 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:03.746485 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:03.746454 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:03.887905 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:03.887868 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz"] Jul 22 09:57:03.894493 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:03.888598 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" podUID="91c30c08-fb9b-45cf-bdc6-f1115ac66394" containerName="istio-proxy" containerID="cri-o://388244e334da0e90fcca4f3801a0620e0742b1a583d339a90884dc94e35619f2" gracePeriod=30 Jul 22 09:57:03.907447 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:03.907381 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:04.181586 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.181506 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:04.477363 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.477257 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:04.754173 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.754097 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv"] Jul 22 09:57:04.754529 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.754417 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dafb95c1-2950-41fc-a94e-3225ca068664" containerName="istio-proxy" Jul 22 09:57:04.754529 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.754434 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="dafb95c1-2950-41fc-a94e-3225ca068664" containerName="istio-proxy" Jul 22 09:57:04.754529 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.754446 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fb5319a-57b4-4ad2-abfb-e30f19748d79" containerName="maas-api" Jul 22 09:57:04.754529 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.754455 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fb5319a-57b4-4ad2-abfb-e30f19748d79" containerName="maas-api" Jul 22 09:57:04.754529 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.754466 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1c9952a-d4ab-4a63-875d-c247a58dea81" containerName="maas-api" Jul 22 09:57:04.754529 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.754475 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1c9952a-d4ab-4a63-875d-c247a58dea81" containerName="maas-api" Jul 22 09:57:04.754529 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.754490 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dafb95c1-2950-41fc-a94e-3225ca068664" containerName="istio-proxy" Jul 22 09:57:04.754529 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.754496 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="dafb95c1-2950-41fc-a94e-3225ca068664" containerName="istio-proxy" Jul 22 09:57:04.754773 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.754553 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="dafb95c1-2950-41fc-a94e-3225ca068664" containerName="istio-proxy" Jul 22 09:57:04.754773 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.754560 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1c9952a-d4ab-4a63-875d-c247a58dea81" containerName="maas-api" Jul 22 09:57:04.754773 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.754568 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fb5319a-57b4-4ad2-abfb-e30f19748d79" containerName="maas-api" Jul 22 09:57:04.757543 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.757525 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.760451 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.760427 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-99612b-openshift-default-dockercfg-7255d\"" Jul 22 09:57:04.771340 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.771289 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv"] Jul 22 09:57:04.782984 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.782956 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:04.880481 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.880444 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-workload-socket\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.880662 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.880492 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istiod-ca-cert\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.880662 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.880544 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnf2z\" (UniqueName: \"kubernetes.io/projected/0b6771b4-2f19-494c-b3cc-fff89d187b2b-kube-api-access-vnf2z\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.880662 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.880591 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-workload-certs\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.880662 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.880609 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-token\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.880662 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.880624 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-envoy\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.880662 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.880639 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-data\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.880662 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.880654 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-podinfo\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.880887 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.880724 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-credential-socket\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.972877 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.972841 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:04.981978 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.981954 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-workload-certs\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.982078 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.981987 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-token\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.982078 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.982004 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-envoy\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.982078 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.982019 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-data\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.982078 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.982036 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-podinfo\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.982271 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.982183 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-credential-socket\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.982351 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.982276 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-workload-socket\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.982417 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.982349 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istiod-ca-cert\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.982417 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.982379 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnf2z\" (UniqueName: \"kubernetes.io/projected/0b6771b4-2f19-494c-b3cc-fff89d187b2b-kube-api-access-vnf2z\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.982530 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.982438 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-workload-certs\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.982585 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.982549 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-data\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.982707 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.982688 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-workload-socket\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.982762 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.982702 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-credential-socket\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.982977 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.982960 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istiod-ca-cert\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.984459 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.984440 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-envoy\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.984585 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.984569 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-podinfo\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.989839 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.989815 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-token\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:04.990032 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:04.990010 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnf2z\" (UniqueName: \"kubernetes.io/projected/0b6771b4-2f19-494c-b3cc-fff89d187b2b-kube-api-access-vnf2z\") pod \"e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:05.067837 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:05.067801 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:05.218671 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:05.218646 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv"] Jul 22 09:57:05.220180 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:57:05.220157 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6771b4_2f19_494c_b3cc_fff89d187b2b.slice/crio-f2d8e6e3ea434769248bd842fa211ea3b952c7a3975ce6db42066dbe29f53f17 WatchSource:0}: Error finding container f2d8e6e3ea434769248bd842fa211ea3b952c7a3975ce6db42066dbe29f53f17: Status 404 returned error can't find the container with id f2d8e6e3ea434769248bd842fa211ea3b952c7a3975ce6db42066dbe29f53f17 Jul 22 09:57:05.222765 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:05.222734 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:57:05.222844 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:05.222799 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:57:05.222844 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:05.222826 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:57:05.658675 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:05.658644 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" event={"ID":"0b6771b4-2f19-494c-b3cc-fff89d187b2b","Type":"ContainerStarted","Data":"b53811b8fa4f9fff886006d2244838a0db8498d81f2056fd7d2c56d39bc5592a"} Jul 22 09:57:05.658859 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:05.658708 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" event={"ID":"0b6771b4-2f19-494c-b3cc-fff89d187b2b","Type":"ContainerStarted","Data":"f2d8e6e3ea434769248bd842fa211ea3b952c7a3975ce6db42066dbe29f53f17"} Jul 22 09:57:05.682735 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:05.682685 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" podStartSLOduration=1.682669691 podStartE2EDuration="1.682669691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:57:05.679448142 +0000 UTC m=+2356.143242583" watchObservedRunningTime="2026-07-22 09:57:05.682669691 +0000 UTC m=+2356.146464129" Jul 22 09:57:05.862073 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:05.862040 2549 patch_prober.go:28] interesting pod/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 09:57:05.862488 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:05.862101 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" podUID="91c30c08-fb9b-45cf-bdc6-f1115ac66394" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 09:57:06.068654 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:06.068623 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:06.073556 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:06.073532 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:06.664016 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:06.663988 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:06.664860 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:06.664841 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:07.598327 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:07.598267 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:07.693339 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:07.693290 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8"] Jul 22 09:57:07.693705 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:07.693693 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="dafb95c1-2950-41fc-a94e-3225ca068664" containerName="istio-proxy" Jul 22 09:57:07.694861 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:07.694836 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:07.696581 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:07.696562 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8" Jul 22 09:57:07.700164 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:07.700144 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-876r5\"" Jul 22 09:57:07.700426 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:07.700410 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-99612b-serving-cert\"" Jul 22 09:57:07.709654 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:07.709633 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8"] Jul 22 09:57:07.768239 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:07.768208 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:07.808421 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:07.808369 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c77pv\" (UniqueName: \"kubernetes.io/projected/c980835c-4818-4404-b928-afbda5084a6b-kube-api-access-c77pv\") pod \"maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8\" (UID: \"c980835c-4818-4404-b928-afbda5084a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8" Jul 22 09:57:07.808610 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:07.808444 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c980835c-4818-4404-b928-afbda5084a6b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8\" (UID: \"c980835c-4818-4404-b928-afbda5084a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8" Jul 22 09:57:07.808670 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:07.808618 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c980835c-4818-4404-b928-afbda5084a6b-maas-api-tls\") pod \"maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8\" (UID: \"c980835c-4818-4404-b928-afbda5084a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8" Jul 22 09:57:07.867175 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:07.867065 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:07.909060 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:07.909022 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c980835c-4818-4404-b928-afbda5084a6b-maas-api-tls\") pod \"maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8\" (UID: \"c980835c-4818-4404-b928-afbda5084a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8" Jul 22 09:57:07.909224 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:07.909109 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c77pv\" (UniqueName: \"kubernetes.io/projected/c980835c-4818-4404-b928-afbda5084a6b-kube-api-access-c77pv\") pod \"maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8\" (UID: \"c980835c-4818-4404-b928-afbda5084a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8" Jul 22 09:57:07.909224 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:07.909149 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c980835c-4818-4404-b928-afbda5084a6b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8\" (UID: \"c980835c-4818-4404-b928-afbda5084a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8" Jul 22 09:57:07.910062 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:07.910036 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c980835c-4818-4404-b928-afbda5084a6b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8\" (UID: \"c980835c-4818-4404-b928-afbda5084a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8" Jul 22 09:57:07.911629 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:07.911607 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c980835c-4818-4404-b928-afbda5084a6b-maas-api-tls\") pod \"maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8\" (UID: \"c980835c-4818-4404-b928-afbda5084a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8" Jul 22 09:57:07.922530 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:07.922510 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c77pv\" (UniqueName: \"kubernetes.io/projected/c980835c-4818-4404-b928-afbda5084a6b-kube-api-access-c77pv\") pod \"maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8\" (UID: \"c980835c-4818-4404-b928-afbda5084a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8" Jul 22 09:57:08.007470 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:08.007433 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8" Jul 22 09:57:08.150285 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:08.150209 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8"] Jul 22 09:57:08.153425 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:57:08.153394 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc980835c_4818_4404_b928_afbda5084a6b.slice/crio-cb353001747f93d5f279aaa46e53712a0e2551b5c306128ce9ed83e82dc0db3a WatchSource:0}: Error finding container cb353001747f93d5f279aaa46e53712a0e2551b5c306128ce9ed83e82dc0db3a: Status 404 returned error can't find the container with id cb353001747f93d5f279aaa46e53712a0e2551b5c306128ce9ed83e82dc0db3a Jul 22 09:57:08.452083 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:08.451991 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:08.553088 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:08.553052 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:08.676859 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:08.676822 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8" event={"ID":"c980835c-4818-4404-b928-afbda5084a6b","Type":"ContainerStarted","Data":"722051d7b8a8eeea91d263f248f5eafa8aa36655fa5ef0911f4e459dc11eca41"} Jul 22 09:57:08.677325 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:08.676886 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8" event={"ID":"c980835c-4818-4404-b928-afbda5084a6b","Type":"ContainerStarted","Data":"cb353001747f93d5f279aaa46e53712a0e2551b5c306128ce9ed83e82dc0db3a"} Jul 22 09:57:08.677325 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:08.676916 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8" Jul 22 09:57:08.699534 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:08.699484 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8" podStartSLOduration=1.2760908020000001 podStartE2EDuration="1.699468522s" totalImagesPullingTime="423.37772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:57:08.698201374 +0000 UTC m=+2359.161995822" watchObservedRunningTime="2026-07-22 09:57:08.699468522 +0000 UTC m=+2359.163262960" Jul 22 09:57:09.154347 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.154285 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:57:09.322022 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.321986 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-token\") pod \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " Jul 22 09:57:09.322219 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.322054 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-podinfo\") pod \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " Jul 22 09:57:09.322219 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.322080 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-workload-certs\") pod \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " Jul 22 09:57:09.322219 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.322107 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-data\") pod \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " Jul 22 09:57:09.322219 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.322137 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-workload-socket\") pod \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " Jul 22 09:57:09.322219 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.322171 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istiod-ca-cert\") pod \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " Jul 22 09:57:09.322524 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.322225 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhr9h\" (UniqueName: \"kubernetes.io/projected/91c30c08-fb9b-45cf-bdc6-f1115ac66394-kube-api-access-jhr9h\") pod \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " Jul 22 09:57:09.322524 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.322248 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-credential-socket\") pod \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " Jul 22 09:57:09.322524 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.322298 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-envoy\") pod \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\" (UID: \"91c30c08-fb9b-45cf-bdc6-f1115ac66394\") " Jul 22 09:57:09.322524 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.322365 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "91c30c08-fb9b-45cf-bdc6-f1115ac66394" (UID: "91c30c08-fb9b-45cf-bdc6-f1115ac66394"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:57:09.322773 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.322581 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:57:09.322773 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.322661 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "91c30c08-fb9b-45cf-bdc6-f1115ac66394" (UID: "91c30c08-fb9b-45cf-bdc6-f1115ac66394"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:57:09.322773 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.322684 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "91c30c08-fb9b-45cf-bdc6-f1115ac66394" (UID: "91c30c08-fb9b-45cf-bdc6-f1115ac66394"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:57:09.322773 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.322728 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "91c30c08-fb9b-45cf-bdc6-f1115ac66394" (UID: "91c30c08-fb9b-45cf-bdc6-f1115ac66394"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:57:09.323376 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.323346 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-data" (OuterVolumeSpecName: "istio-data") pod "91c30c08-fb9b-45cf-bdc6-f1115ac66394" (UID: "91c30c08-fb9b-45cf-bdc6-f1115ac66394"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:57:09.324513 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.324486 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-token" (OuterVolumeSpecName: "istio-token") pod "91c30c08-fb9b-45cf-bdc6-f1115ac66394" (UID: "91c30c08-fb9b-45cf-bdc6-f1115ac66394"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:57:09.324513 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.324499 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c30c08-fb9b-45cf-bdc6-f1115ac66394-kube-api-access-jhr9h" (OuterVolumeSpecName: "kube-api-access-jhr9h") pod "91c30c08-fb9b-45cf-bdc6-f1115ac66394" (UID: "91c30c08-fb9b-45cf-bdc6-f1115ac66394"). InnerVolumeSpecName "kube-api-access-jhr9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:57:09.324638 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.324588 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "91c30c08-fb9b-45cf-bdc6-f1115ac66394" (UID: "91c30c08-fb9b-45cf-bdc6-f1115ac66394"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 09:57:09.324638 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.324591 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "91c30c08-fb9b-45cf-bdc6-f1115ac66394" (UID: "91c30c08-fb9b-45cf-bdc6-f1115ac66394"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:57:09.423424 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.423297 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:57:09.423424 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.423365 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:57:09.423424 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.423374 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:57:09.423424 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.423386 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:57:09.423424 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.423400 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:57:09.423424 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.423410 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhr9h\" (UniqueName: \"kubernetes.io/projected/91c30c08-fb9b-45cf-bdc6-f1115ac66394-kube-api-access-jhr9h\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:57:09.423424 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.423422 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:57:09.423424 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.423430 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c30c08-fb9b-45cf-bdc6-f1115ac66394-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 09:57:09.683480 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.683394 2549 generic.go:356] "Generic (PLEG): container finished" podID="91c30c08-fb9b-45cf-bdc6-f1115ac66394" containerID="388244e334da0e90fcca4f3801a0620e0742b1a583d339a90884dc94e35619f2" exitCode=0 Jul 22 09:57:09.683480 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.683439 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" Jul 22 09:57:09.683924 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.683485 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" event={"ID":"91c30c08-fb9b-45cf-bdc6-f1115ac66394","Type":"ContainerDied","Data":"388244e334da0e90fcca4f3801a0620e0742b1a583d339a90884dc94e35619f2"} Jul 22 09:57:09.683924 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.683536 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz" event={"ID":"91c30c08-fb9b-45cf-bdc6-f1115ac66394","Type":"ContainerDied","Data":"ea6d41f38694edfa937c0bb2d9621b76c757a87a595ce64bfae8f2f200e2d42e"} Jul 22 09:57:09.683924 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.683557 2549 scope.go:117] "RemoveContainer" containerID="388244e334da0e90fcca4f3801a0620e0742b1a583d339a90884dc94e35619f2" Jul 22 09:57:09.692847 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.692829 2549 scope.go:117] "RemoveContainer" containerID="0317d270f2ab419a7021307754f8bc347433982a6e1b75b9d83c74654a0fe76c" Jul 22 09:57:09.700891 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.700875 2549 scope.go:117] "RemoveContainer" containerID="388244e334da0e90fcca4f3801a0620e0742b1a583d339a90884dc94e35619f2" Jul 22 09:57:09.701142 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:57:09.701124 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"388244e334da0e90fcca4f3801a0620e0742b1a583d339a90884dc94e35619f2\": container with ID starting with 388244e334da0e90fcca4f3801a0620e0742b1a583d339a90884dc94e35619f2 not found: ID does not exist" containerID="388244e334da0e90fcca4f3801a0620e0742b1a583d339a90884dc94e35619f2" Jul 22 09:57:09.701184 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.701150 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"388244e334da0e90fcca4f3801a0620e0742b1a583d339a90884dc94e35619f2"} err="failed to get container status \"388244e334da0e90fcca4f3801a0620e0742b1a583d339a90884dc94e35619f2\": rpc error: code = NotFound desc = could not find container \"388244e334da0e90fcca4f3801a0620e0742b1a583d339a90884dc94e35619f2\": container with ID starting with 388244e334da0e90fcca4f3801a0620e0742b1a583d339a90884dc94e35619f2 not found: ID does not exist" Jul 22 09:57:09.701184 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.701166 2549 scope.go:117] "RemoveContainer" containerID="0317d270f2ab419a7021307754f8bc347433982a6e1b75b9d83c74654a0fe76c" Jul 22 09:57:09.701435 ip-10-0-138-138 kubenswrapper[2549]: E0722 09:57:09.701416 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0317d270f2ab419a7021307754f8bc347433982a6e1b75b9d83c74654a0fe76c\": container with ID starting with 0317d270f2ab419a7021307754f8bc347433982a6e1b75b9d83c74654a0fe76c not found: ID does not exist" containerID="0317d270f2ab419a7021307754f8bc347433982a6e1b75b9d83c74654a0fe76c" Jul 22 09:57:09.701484 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.701442 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0317d270f2ab419a7021307754f8bc347433982a6e1b75b9d83c74654a0fe76c"} err="failed to get container status \"0317d270f2ab419a7021307754f8bc347433982a6e1b75b9d83c74654a0fe76c\": rpc error: code = NotFound desc = could not find container \"0317d270f2ab419a7021307754f8bc347433982a6e1b75b9d83c74654a0fe76c\": container with ID starting with 0317d270f2ab419a7021307754f8bc347433982a6e1b75b9d83c74654a0fe76c not found: ID does not exist" Jul 22 09:57:09.706196 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.706173 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz"] Jul 22 09:57:09.709948 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:09.709923 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a5acdb-openshift-default-7d4b7d46b9-wp2fz"] Jul 22 09:57:10.340224 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:10.340194 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91c30c08-fb9b-45cf-bdc6-f1115ac66394" path="/var/lib/kubelet/pods/91c30c08-fb9b-45cf-bdc6-f1115ac66394/volumes" Jul 22 09:57:12.160616 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.160583 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:12.503123 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.503025 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:12.807180 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.807144 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt"] Jul 22 09:57:12.807671 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.807644 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91c30c08-fb9b-45cf-bdc6-f1115ac66394" containerName="istio-proxy" Jul 22 09:57:12.807671 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.807666 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="91c30c08-fb9b-45cf-bdc6-f1115ac66394" containerName="istio-proxy" Jul 22 09:57:12.807846 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.807683 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91c30c08-fb9b-45cf-bdc6-f1115ac66394" containerName="istio-proxy" Jul 22 09:57:12.807846 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.807699 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="91c30c08-fb9b-45cf-bdc6-f1115ac66394" containerName="istio-proxy" Jul 22 09:57:12.807846 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.807802 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="91c30c08-fb9b-45cf-bdc6-f1115ac66394" containerName="istio-proxy" Jul 22 09:57:12.807846 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.807812 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="91c30c08-fb9b-45cf-bdc6-f1115ac66394" containerName="istio-proxy" Jul 22 09:57:12.812256 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.812238 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.814767 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.814748 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-88ceff-openshift-default-dockercfg-grjgd\"" Jul 22 09:57:12.822645 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.822623 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt"] Jul 22 09:57:12.833947 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.833919 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:12.850627 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.850600 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-workload-certs\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.850750 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.850647 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-envoy\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.850750 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.850688 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-podinfo\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.850750 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.850715 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e99b786-a54d-45bd-830e-d10b40c7b40b-istiod-ca-cert\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.850856 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.850748 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-credential-socket\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.850856 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.850768 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-data\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.850856 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.850782 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-token\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.850856 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.850811 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-workload-socket\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.850856 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.850845 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfbfd\" (UniqueName: \"kubernetes.io/projected/0e99b786-a54d-45bd-830e-d10b40c7b40b-kube-api-access-hfbfd\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.951737 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.951697 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-workload-socket\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.951935 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.951758 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfbfd\" (UniqueName: \"kubernetes.io/projected/0e99b786-a54d-45bd-830e-d10b40c7b40b-kube-api-access-hfbfd\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.951935 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.951792 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-workload-certs\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.951935 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.951843 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-envoy\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.951935 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.951870 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-podinfo\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.951935 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.951905 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e99b786-a54d-45bd-830e-d10b40c7b40b-istiod-ca-cert\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.952207 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.951937 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-credential-socket\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.952207 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.952144 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-data\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.952207 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.952161 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-workload-socket\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.952207 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.952181 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-token\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.952456 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.952213 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-workload-certs\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.952456 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.952441 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-credential-socket\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.952553 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.952532 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-data\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.952667 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.952649 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e99b786-a54d-45bd-830e-d10b40c7b40b-istiod-ca-cert\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.954193 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.954176 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-envoy\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.954355 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.954339 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-podinfo\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.960125 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.960100 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfbfd\" (UniqueName: \"kubernetes.io/projected/0e99b786-a54d-45bd-830e-d10b40c7b40b-kube-api-access-hfbfd\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.960239 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.960187 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-token\") pod \"e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:12.980540 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:12.980513 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:13.123522 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:13.123429 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:13.269618 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:13.269587 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt"] Jul 22 09:57:13.271360 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:57:13.271333 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e99b786_a54d_45bd_830e_d10b40c7b40b.slice/crio-8b475abf6a79efff5823c4fc60a5d4bc8c355b3ee54ae4077a6a229559437589 WatchSource:0}: Error finding container 8b475abf6a79efff5823c4fc60a5d4bc8c355b3ee54ae4077a6a229559437589: Status 404 returned error can't find the container with id 8b475abf6a79efff5823c4fc60a5d4bc8c355b3ee54ae4077a6a229559437589 Jul 22 09:57:13.273589 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:13.273551 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:57:13.273677 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:13.273627 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:57:13.273677 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:13.273657 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:57:13.706505 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:13.706469 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" event={"ID":"0e99b786-a54d-45bd-830e-d10b40c7b40b","Type":"ContainerStarted","Data":"4bc4e58cc4d8f8d6a8d82f2cfdf16ca14aea1bbe2434dc14269776ff3436079a"} Jul 22 09:57:13.706696 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:13.706529 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" event={"ID":"0e99b786-a54d-45bd-830e-d10b40c7b40b","Type":"ContainerStarted","Data":"8b475abf6a79efff5823c4fc60a5d4bc8c355b3ee54ae4077a6a229559437589"} Jul 22 09:57:13.731466 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:13.731414 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" podStartSLOduration=1.731398212 podStartE2EDuration="1.731398212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:57:13.729412487 +0000 UTC m=+2364.193206925" watchObservedRunningTime="2026-07-22 09:57:13.731398212 +0000 UTC m=+2364.195192649" Jul 22 09:57:14.124197 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:14.124162 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:14.129177 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:14.129152 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:14.689185 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:14.689158 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8" Jul 22 09:57:14.711168 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:14.711146 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:14.711966 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:14.711950 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:15.871669 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:15.871634 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf"] Jul 22 09:57:15.875384 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:15.875361 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf" Jul 22 09:57:15.877836 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:15.877813 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-88ceff-serving-cert\"" Jul 22 09:57:15.887592 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:15.887569 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf"] Jul 22 09:57:15.938432 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:15.938396 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:15.972632 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:15.972602 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7vwf\" (UniqueName: \"kubernetes.io/projected/54d27d0d-ef7b-4e41-b3be-e67394ecd43f-kube-api-access-x7vwf\") pod \"maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf\" (UID: \"54d27d0d-ef7b-4e41-b3be-e67394ecd43f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf" Jul 22 09:57:15.972766 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:15.972652 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d27d0d-ef7b-4e41-b3be-e67394ecd43f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf\" (UID: \"54d27d0d-ef7b-4e41-b3be-e67394ecd43f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf" Jul 22 09:57:15.972766 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:15.972715 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54d27d0d-ef7b-4e41-b3be-e67394ecd43f-maas-api-tls\") pod \"maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf\" (UID: \"54d27d0d-ef7b-4e41-b3be-e67394ecd43f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf" Jul 22 09:57:16.038825 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:16.038799 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:16.073440 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:16.073387 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54d27d0d-ef7b-4e41-b3be-e67394ecd43f-maas-api-tls\") pod \"maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf\" (UID: \"54d27d0d-ef7b-4e41-b3be-e67394ecd43f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf" Jul 22 09:57:16.073593 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:16.073489 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7vwf\" (UniqueName: \"kubernetes.io/projected/54d27d0d-ef7b-4e41-b3be-e67394ecd43f-kube-api-access-x7vwf\") pod \"maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf\" (UID: \"54d27d0d-ef7b-4e41-b3be-e67394ecd43f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf" Jul 22 09:57:16.073593 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:16.073536 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d27d0d-ef7b-4e41-b3be-e67394ecd43f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf\" (UID: \"54d27d0d-ef7b-4e41-b3be-e67394ecd43f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf" Jul 22 09:57:16.074222 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:16.074198 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d27d0d-ef7b-4e41-b3be-e67394ecd43f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf\" (UID: \"54d27d0d-ef7b-4e41-b3be-e67394ecd43f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf" Jul 22 09:57:16.075851 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:16.075832 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54d27d0d-ef7b-4e41-b3be-e67394ecd43f-maas-api-tls\") pod \"maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf\" (UID: \"54d27d0d-ef7b-4e41-b3be-e67394ecd43f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf" Jul 22 09:57:16.081447 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:16.081425 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7vwf\" (UniqueName: \"kubernetes.io/projected/54d27d0d-ef7b-4e41-b3be-e67394ecd43f-kube-api-access-x7vwf\") pod \"maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf\" (UID: \"54d27d0d-ef7b-4e41-b3be-e67394ecd43f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf" Jul 22 09:57:16.161323 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:16.161230 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:16.187123 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:16.187089 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf" Jul 22 09:57:16.361713 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:16.361688 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf"] Jul 22 09:57:16.389252 ip-10-0-138-138 kubenswrapper[2549]: W0722 09:57:16.389217 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d27d0d_ef7b_4e41_b3be_e67394ecd43f.slice/crio-64d769ad502fa27a0353b7aeb85039eda83f04b2d9588753048a3d573f3d1ea7 WatchSource:0}: Error finding container 64d769ad502fa27a0353b7aeb85039eda83f04b2d9588753048a3d573f3d1ea7: Status 404 returned error can't find the container with id 64d769ad502fa27a0353b7aeb85039eda83f04b2d9588753048a3d573f3d1ea7 Jul 22 09:57:16.427155 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:16.427092 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:16.700562 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:16.700456 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:16.722159 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:16.722128 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf" event={"ID":"54d27d0d-ef7b-4e41-b3be-e67394ecd43f","Type":"ContainerStarted","Data":"64d769ad502fa27a0353b7aeb85039eda83f04b2d9588753048a3d573f3d1ea7"} Jul 22 09:57:17.728616 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:17.728580 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf" event={"ID":"54d27d0d-ef7b-4e41-b3be-e67394ecd43f","Type":"ContainerStarted","Data":"3566061588efe3f033b1f581ba9c0258d441e5726ce2345babd5ecac090a44a9"} Jul 22 09:57:17.728997 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:17.728714 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf" Jul 22 09:57:17.754616 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:17.754560 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf" podStartSLOduration=2.257714869 podStartE2EDuration="2.754545209s" totalImagesPullingTime="496.83034ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:57:17.753510729 +0000 UTC m=+2368.217305168" watchObservedRunningTime="2026-07-22 09:57:17.754545209 +0000 UTC m=+2368.218339646" Jul 22 09:57:20.229539 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:20.229508 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:22.706547 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:22.706497 2549 patch_prober.go:28] interesting pod/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 22 09:57:22.707036 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:22.706580 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" podUID="0b6771b4-2f19-494c-b3cc-fff89d187b2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 22 09:57:23.739382 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:23.739354 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf" Jul 22 09:57:23.766396 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:23.763703 2549 generic.go:356] "Generic (PLEG): container finished" podID="0b6771b4-2f19-494c-b3cc-fff89d187b2b" containerID="b53811b8fa4f9fff886006d2244838a0db8498d81f2056fd7d2c56d39bc5592a" exitCode=0 Jul 22 09:57:23.766396 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:23.763841 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" event={"ID":"0b6771b4-2f19-494c-b3cc-fff89d187b2b","Type":"ContainerDied","Data":"b53811b8fa4f9fff886006d2244838a0db8498d81f2056fd7d2c56d39bc5592a"} Jul 22 09:57:23.766396 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:23.764285 2549 scope.go:117] "RemoveContainer" containerID="b53811b8fa4f9fff886006d2244838a0db8498d81f2056fd7d2c56d39bc5592a" Jul 22 09:57:23.766396 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:23.765700 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:57:23.766396 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:23.765780 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:57:23.766396 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:23.765826 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:57:24.775209 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:24.775161 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" event={"ID":"0b6771b4-2f19-494c-b3cc-fff89d187b2b","Type":"ContainerStarted","Data":"caae911a5eda743304ccd0089c630f153b1b7eb34417380a7f710624c0ec3483"} Jul 22 09:57:25.068784 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:25.068741 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:25.069224 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:25.069193 2549 patch_prober.go:28] interesting pod/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 22 09:57:25.069331 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:25.069263 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" podUID="0b6771b4-2f19-494c-b3cc-fff89d187b2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 22 09:57:26.069627 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:26.069590 2549 patch_prober.go:28] interesting pod/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 22 09:57:26.070242 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:26.070193 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" podUID="0b6771b4-2f19-494c-b3cc-fff89d187b2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 22 09:57:27.069252 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:27.069206 2549 patch_prober.go:28] interesting pod/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 22 09:57:27.069471 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:27.069275 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" podUID="0b6771b4-2f19-494c-b3cc-fff89d187b2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 22 09:57:28.069202 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:28.069155 2549 patch_prober.go:28] interesting pod/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 22 09:57:28.069699 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:28.069229 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" podUID="0b6771b4-2f19-494c-b3cc-fff89d187b2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 22 09:57:29.071849 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:29.071818 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:29.072326 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:29.072208 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:29.072970 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:29.072954 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 09:57:29.228019 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:29.227977 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:29.332597 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:29.332509 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:29.680620 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:29.680540 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:29.803051 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:29.802259 2549 patch_prober.go:28] interesting pod/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": read tcp 10.133.0.2:51908->10.133.0.60:15021: read: connection reset by peer" start-of-body= Jul 22 09:57:29.803051 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:29.802354 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" podUID="0e99b786-a54d-45bd-830e-d10b40c7b40b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": read tcp 10.133.0.2:51908->10.133.0.60:15021: read: connection reset by peer" Jul 22 09:57:30.815648 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:30.815614 2549 generic.go:356] "Generic (PLEG): container finished" podID="0e99b786-a54d-45bd-830e-d10b40c7b40b" containerID="4bc4e58cc4d8f8d6a8d82f2cfdf16ca14aea1bbe2434dc14269776ff3436079a" exitCode=0 Jul 22 09:57:30.816029 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:30.815689 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" event={"ID":"0e99b786-a54d-45bd-830e-d10b40c7b40b","Type":"ContainerDied","Data":"4bc4e58cc4d8f8d6a8d82f2cfdf16ca14aea1bbe2434dc14269776ff3436079a"} Jul 22 09:57:30.816222 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:30.816206 2549 scope.go:117] "RemoveContainer" containerID="4bc4e58cc4d8f8d6a8d82f2cfdf16ca14aea1bbe2434dc14269776ff3436079a" Jul 22 09:57:30.817210 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:30.817185 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:57:30.817274 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:30.817249 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:57:30.817333 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:30.817299 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 09:57:31.824458 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:31.824415 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" event={"ID":"0e99b786-a54d-45bd-830e-d10b40c7b40b","Type":"ContainerStarted","Data":"73c03cea58704809bfec14b7830b9a471126b44c88006e600b049568f6c7cb5b"} Jul 22 09:57:32.124533 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:32.124444 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:32.124763 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:32.124741 2549 patch_prober.go:28] interesting pod/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 22 09:57:32.124817 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:32.124790 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" podUID="0e99b786-a54d-45bd-830e-d10b40c7b40b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 22 09:57:33.123978 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:33.123940 2549 patch_prober.go:28] interesting pod/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 22 09:57:33.124496 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:33.124004 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" podUID="0e99b786-a54d-45bd-830e-d10b40c7b40b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 22 09:57:33.697160 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:33.697122 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:33.814423 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:33.814394 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:34.123987 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:34.123947 2549 patch_prober.go:28] interesting pod/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 22 09:57:34.124470 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:34.124037 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" podUID="0e99b786-a54d-45bd-830e-d10b40c7b40b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 22 09:57:35.124429 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:35.124395 2549 patch_prober.go:28] interesting pod/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 22 09:57:35.124836 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:35.124453 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" podUID="0e99b786-a54d-45bd-830e-d10b40c7b40b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 22 09:57:36.135884 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:36.135847 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:36.136420 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:36.136257 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:36.137035 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:36.137018 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 09:57:37.200567 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:37.200532 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:37.302410 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:37.302370 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:48.070421 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:48.070387 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:57:50.445462 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:50.445435 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcsgw_dc8d8789-74a0-44ec-8902-44a9650c41ac/ovn-acl-logging/0.log" Jul 22 09:57:50.464135 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:50.464115 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcsgw_dc8d8789-74a0-44ec-8902-44a9650c41ac/ovn-acl-logging/0.log" Jul 22 09:57:50.584583 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:57:50.584554 2549 scope.go:117] "RemoveContainer" containerID="bcdba5f6604f44f657721a1a3ac9d9ffeca276f6959293e315899e975adc6096" Jul 22 09:58:45.720383 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:58:45.720334 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 09:59:59.317704 ip-10-0-138-138 kubenswrapper[2549]: I0722 09:59:59.317666 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:00:00.032671 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:00:00.032637 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:00:00.377086 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:00:00.377006 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:00:00.496890 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:00:00.496854 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:00:21.710908 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:00:21.710870 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:00:33.019835 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:00:33.019793 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:00:55.286923 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:00:55.286884 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:01:41.187534 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:01:41.187456 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:01:42.098142 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:01:42.098103 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:01:50.318476 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:01:50.318441 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:02:15.841008 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:02:15.840972 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:02:30.072647 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:02:30.072598 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:02:30.174458 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:02:30.174423 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:02:34.545078 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:02:34.545045 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:02:34.646017 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:02:34.645988 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:02:38.231988 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:02:38.231954 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:02:38.336957 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:02:38.336923 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:02:50.483243 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:02:50.483176 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcsgw_dc8d8789-74a0-44ec-8902-44a9650c41ac/ovn-acl-logging/0.log" Jul 22 10:02:50.500988 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:02:50.500968 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcsgw_dc8d8789-74a0-44ec-8902-44a9650c41ac/ovn-acl-logging/0.log" Jul 22 10:02:58.573252 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:02:58.573203 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:02:59.410999 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:02:59.410965 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:03:19.478716 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:19.478684 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:03:19.578983 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:19.578947 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:03:29.220322 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:29.220263 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8"] Jul 22 10:03:29.220915 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:29.220608 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8" podUID="c980835c-4818-4404-b928-afbda5084a6b" containerName="maas-api" containerID="cri-o://722051d7b8a8eeea91d263f248f5eafa8aa36655fa5ef0911f4e459dc11eca41" gracePeriod=30 Jul 22 10:03:29.248222 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:29.248195 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:03:29.472686 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:29.472632 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8" Jul 22 10:03:29.543845 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:29.543814 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c77pv\" (UniqueName: \"kubernetes.io/projected/c980835c-4818-4404-b928-afbda5084a6b-kube-api-access-c77pv\") pod \"c980835c-4818-4404-b928-afbda5084a6b\" (UID: \"c980835c-4818-4404-b928-afbda5084a6b\") " Jul 22 10:03:29.544024 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:29.543858 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c980835c-4818-4404-b928-afbda5084a6b-maas-api-tls\") pod \"c980835c-4818-4404-b928-afbda5084a6b\" (UID: \"c980835c-4818-4404-b928-afbda5084a6b\") " Jul 22 10:03:29.544024 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:29.543881 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c980835c-4818-4404-b928-afbda5084a6b-openshift-service-ca\") pod \"c980835c-4818-4404-b928-afbda5084a6b\" (UID: \"c980835c-4818-4404-b928-afbda5084a6b\") " Jul 22 10:03:29.544295 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:29.544261 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c980835c-4818-4404-b928-afbda5084a6b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c980835c-4818-4404-b928-afbda5084a6b" (UID: "c980835c-4818-4404-b928-afbda5084a6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 10:03:29.545910 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:29.545883 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c980835c-4818-4404-b928-afbda5084a6b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c980835c-4818-4404-b928-afbda5084a6b" (UID: "c980835c-4818-4404-b928-afbda5084a6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 10:03:29.545973 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:29.545912 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c980835c-4818-4404-b928-afbda5084a6b-kube-api-access-c77pv" (OuterVolumeSpecName: "kube-api-access-c77pv") pod "c980835c-4818-4404-b928-afbda5084a6b" (UID: "c980835c-4818-4404-b928-afbda5084a6b"). InnerVolumeSpecName "kube-api-access-c77pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 10:03:29.614564 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:29.614527 2549 generic.go:356] "Generic (PLEG): container finished" podID="c980835c-4818-4404-b928-afbda5084a6b" containerID="722051d7b8a8eeea91d263f248f5eafa8aa36655fa5ef0911f4e459dc11eca41" exitCode=0 Jul 22 10:03:29.614710 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:29.614574 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8" Jul 22 10:03:29.614710 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:29.614616 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8" event={"ID":"c980835c-4818-4404-b928-afbda5084a6b","Type":"ContainerDied","Data":"722051d7b8a8eeea91d263f248f5eafa8aa36655fa5ef0911f4e459dc11eca41"} Jul 22 10:03:29.614710 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:29.614672 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8" event={"ID":"c980835c-4818-4404-b928-afbda5084a6b","Type":"ContainerDied","Data":"cb353001747f93d5f279aaa46e53712a0e2551b5c306128ce9ed83e82dc0db3a"} Jul 22 10:03:29.614710 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:29.614701 2549 scope.go:117] "RemoveContainer" containerID="722051d7b8a8eeea91d263f248f5eafa8aa36655fa5ef0911f4e459dc11eca41" Jul 22 10:03:29.623476 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:29.623460 2549 scope.go:117] "RemoveContainer" containerID="722051d7b8a8eeea91d263f248f5eafa8aa36655fa5ef0911f4e459dc11eca41" Jul 22 10:03:29.623732 ip-10-0-138-138 kubenswrapper[2549]: E0722 10:03:29.623713 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"722051d7b8a8eeea91d263f248f5eafa8aa36655fa5ef0911f4e459dc11eca41\": container with ID starting with 722051d7b8a8eeea91d263f248f5eafa8aa36655fa5ef0911f4e459dc11eca41 not found: ID does not exist" containerID="722051d7b8a8eeea91d263f248f5eafa8aa36655fa5ef0911f4e459dc11eca41" Jul 22 10:03:29.623785 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:29.623741 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"722051d7b8a8eeea91d263f248f5eafa8aa36655fa5ef0911f4e459dc11eca41"} err="failed to get container status \"722051d7b8a8eeea91d263f248f5eafa8aa36655fa5ef0911f4e459dc11eca41\": rpc error: code = NotFound desc = could not find container \"722051d7b8a8eeea91d263f248f5eafa8aa36655fa5ef0911f4e459dc11eca41\": container with ID starting with 722051d7b8a8eeea91d263f248f5eafa8aa36655fa5ef0911f4e459dc11eca41 not found: ID does not exist" Jul 22 10:03:29.636668 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:29.636644 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8"] Jul 22 10:03:29.639825 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:29.639803 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-99612b-5997f46d94-h9qq8"] Jul 22 10:03:29.645017 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:29.644999 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c980835c-4818-4404-b928-afbda5084a6b-maas-api-tls\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:03:29.645079 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:29.645021 2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c980835c-4818-4404-b928-afbda5084a6b-openshift-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:03:29.645079 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:29.645033 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c77pv\" (UniqueName: \"kubernetes.io/projected/c980835c-4818-4404-b928-afbda5084a6b-kube-api-access-c77pv\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:03:30.339265 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:30.339233 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c980835c-4818-4404-b928-afbda5084a6b" path="/var/lib/kubelet/pods/c980835c-4818-4404-b928-afbda5084a6b/volumes" Jul 22 10:03:33.779986 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:33.779951 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:03:34.176554 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:34.176470 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:03:40.422872 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:40.422836 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:03:40.692588 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:40.692503 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv"] Jul 22 10:03:40.692790 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:40.692738 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" podUID="0b6771b4-2f19-494c-b3cc-fff89d187b2b" containerName="istio-proxy" containerID="cri-o://caae911a5eda743304ccd0089c630f153b1b7eb34417380a7f710624c0ec3483" gracePeriod=30 Jul 22 10:03:41.053607 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:41.053576 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:03:41.152477 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:41.152449 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:03:42.449459 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:42.449425 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:03:43.818610 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:43.818578 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:03:44.075068 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:44.074990 2549 patch_prober.go:28] interesting pod/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 10:03:44.075216 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:44.075065 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" podUID="0b6771b4-2f19-494c-b3cc-fff89d187b2b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 10:03:45.947772 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:45.947749 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 10:03:46.084638 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.084607 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-podinfo\") pod \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " Jul 22 10:03:46.084799 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.084651 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-credential-socket\") pod \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " Jul 22 10:03:46.084799 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.084687 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-workload-socket\") pod \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " Jul 22 10:03:46.084890 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.084806 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnf2z\" (UniqueName: \"kubernetes.io/projected/0b6771b4-2f19-494c-b3cc-fff89d187b2b-kube-api-access-vnf2z\") pod \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " Jul 22 10:03:46.084890 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.084871 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istiod-ca-cert\") pod \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " Jul 22 10:03:46.084982 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.084913 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-envoy\") pod \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " Jul 22 10:03:46.084982 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.084940 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0b6771b4-2f19-494c-b3cc-fff89d187b2b" (UID: "0b6771b4-2f19-494c-b3cc-fff89d187b2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:03:46.084982 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.084952 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-token\") pod \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " Jul 22 10:03:46.084982 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.084962 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0b6771b4-2f19-494c-b3cc-fff89d187b2b" (UID: "0b6771b4-2f19-494c-b3cc-fff89d187b2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:03:46.085207 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.084991 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-data\") pod \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " Jul 22 10:03:46.085207 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.085015 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-workload-certs\") pod \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\" (UID: \"0b6771b4-2f19-494c-b3cc-fff89d187b2b\") " Jul 22 10:03:46.085340 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.085265 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:03:46.085340 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.085282 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:03:46.085340 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.085281 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0b6771b4-2f19-494c-b3cc-fff89d187b2b" (UID: "0b6771b4-2f19-494c-b3cc-fff89d187b2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 10:03:46.085480 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.085403 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0b6771b4-2f19-494c-b3cc-fff89d187b2b" (UID: "0b6771b4-2f19-494c-b3cc-fff89d187b2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:03:46.085760 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.085739 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-data" (OuterVolumeSpecName: "istio-data") pod "0b6771b4-2f19-494c-b3cc-fff89d187b2b" (UID: "0b6771b4-2f19-494c-b3cc-fff89d187b2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:03:46.086973 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.086950 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0b6771b4-2f19-494c-b3cc-fff89d187b2b" (UID: "0b6771b4-2f19-494c-b3cc-fff89d187b2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:03:46.087158 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.087134 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b6771b4-2f19-494c-b3cc-fff89d187b2b" (UID: "0b6771b4-2f19-494c-b3cc-fff89d187b2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 10:03:46.087358 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.087333 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0b6771b4-2f19-494c-b3cc-fff89d187b2b" (UID: "0b6771b4-2f19-494c-b3cc-fff89d187b2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 10:03:46.087451 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.087387 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6771b4-2f19-494c-b3cc-fff89d187b2b-kube-api-access-vnf2z" (OuterVolumeSpecName: "kube-api-access-vnf2z") pod "0b6771b4-2f19-494c-b3cc-fff89d187b2b" (UID: "0b6771b4-2f19-494c-b3cc-fff89d187b2b"). InnerVolumeSpecName "kube-api-access-vnf2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 10:03:46.186546 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.186516 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:03:46.186546 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.186543 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:03:46.186546 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.186552 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnf2z\" (UniqueName: \"kubernetes.io/projected/0b6771b4-2f19-494c-b3cc-fff89d187b2b-kube-api-access-vnf2z\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:03:46.186782 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.186564 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:03:46.186782 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.186573 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:03:46.186782 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.186580 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:03:46.186782 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.186587 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6771b4-2f19-494c-b3cc-fff89d187b2b-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:03:46.707658 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.707562 2549 generic.go:356] "Generic (PLEG): container finished" podID="0b6771b4-2f19-494c-b3cc-fff89d187b2b" containerID="caae911a5eda743304ccd0089c630f153b1b7eb34417380a7f710624c0ec3483" exitCode=0 Jul 22 10:03:46.707658 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.707617 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" Jul 22 10:03:46.707658 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.707615 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" event={"ID":"0b6771b4-2f19-494c-b3cc-fff89d187b2b","Type":"ContainerDied","Data":"caae911a5eda743304ccd0089c630f153b1b7eb34417380a7f710624c0ec3483"} Jul 22 10:03:46.707903 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.707671 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv" event={"ID":"0b6771b4-2f19-494c-b3cc-fff89d187b2b","Type":"ContainerDied","Data":"f2d8e6e3ea434769248bd842fa211ea3b952c7a3975ce6db42066dbe29f53f17"} Jul 22 10:03:46.707903 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.707701 2549 scope.go:117] "RemoveContainer" containerID="caae911a5eda743304ccd0089c630f153b1b7eb34417380a7f710624c0ec3483" Jul 22 10:03:46.716188 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.716170 2549 scope.go:117] "RemoveContainer" containerID="b53811b8fa4f9fff886006d2244838a0db8498d81f2056fd7d2c56d39bc5592a" Jul 22 10:03:46.724253 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.724234 2549 scope.go:117] "RemoveContainer" containerID="caae911a5eda743304ccd0089c630f153b1b7eb34417380a7f710624c0ec3483" Jul 22 10:03:46.724507 ip-10-0-138-138 kubenswrapper[2549]: E0722 10:03:46.724490 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caae911a5eda743304ccd0089c630f153b1b7eb34417380a7f710624c0ec3483\": container with ID starting with caae911a5eda743304ccd0089c630f153b1b7eb34417380a7f710624c0ec3483 not found: ID does not exist" containerID="caae911a5eda743304ccd0089c630f153b1b7eb34417380a7f710624c0ec3483" Jul 22 10:03:46.724571 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.724516 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caae911a5eda743304ccd0089c630f153b1b7eb34417380a7f710624c0ec3483"} err="failed to get container status \"caae911a5eda743304ccd0089c630f153b1b7eb34417380a7f710624c0ec3483\": rpc error: code = NotFound desc = could not find container \"caae911a5eda743304ccd0089c630f153b1b7eb34417380a7f710624c0ec3483\": container with ID starting with caae911a5eda743304ccd0089c630f153b1b7eb34417380a7f710624c0ec3483 not found: ID does not exist" Jul 22 10:03:46.724571 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.724533 2549 scope.go:117] "RemoveContainer" containerID="b53811b8fa4f9fff886006d2244838a0db8498d81f2056fd7d2c56d39bc5592a" Jul 22 10:03:46.724730 ip-10-0-138-138 kubenswrapper[2549]: E0722 10:03:46.724716 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b53811b8fa4f9fff886006d2244838a0db8498d81f2056fd7d2c56d39bc5592a\": container with ID starting with b53811b8fa4f9fff886006d2244838a0db8498d81f2056fd7d2c56d39bc5592a not found: ID does not exist" containerID="b53811b8fa4f9fff886006d2244838a0db8498d81f2056fd7d2c56d39bc5592a" Jul 22 10:03:46.724773 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.724731 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b53811b8fa4f9fff886006d2244838a0db8498d81f2056fd7d2c56d39bc5592a"} err="failed to get container status \"b53811b8fa4f9fff886006d2244838a0db8498d81f2056fd7d2c56d39bc5592a\": rpc error: code = NotFound desc = could not find container \"b53811b8fa4f9fff886006d2244838a0db8498d81f2056fd7d2c56d39bc5592a\": container with ID starting with b53811b8fa4f9fff886006d2244838a0db8498d81f2056fd7d2c56d39bc5592a not found: ID does not exist" Jul 22 10:03:46.740106 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.740084 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv"] Jul 22 10:03:46.747424 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:46.747403 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-99612b-openshift-default-5697d6d85f-874gv"] Jul 22 10:03:48.338894 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:48.338865 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b6771b4-2f19-494c-b3cc-fff89d187b2b" path="/var/lib/kubelet/pods/0b6771b4-2f19-494c-b3cc-fff89d187b2b/volumes" Jul 22 10:03:51.577140 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:51.577105 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf"] Jul 22 10:03:51.577582 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:51.577323 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf" podUID="54d27d0d-ef7b-4e41-b3be-e67394ecd43f" containerName="maas-api" containerID="cri-o://3566061588efe3f033b1f581ba9c0258d441e5726ce2345babd5ecac090a44a9" gracePeriod=30 Jul 22 10:03:51.591631 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:51.591603 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:03:51.735883 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:51.735852 2549 generic.go:356] "Generic (PLEG): container finished" podID="54d27d0d-ef7b-4e41-b3be-e67394ecd43f" containerID="3566061588efe3f033b1f581ba9c0258d441e5726ce2345babd5ecac090a44a9" exitCode=0 Jul 22 10:03:51.736050 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:51.735923 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf" event={"ID":"54d27d0d-ef7b-4e41-b3be-e67394ecd43f","Type":"ContainerDied","Data":"3566061588efe3f033b1f581ba9c0258d441e5726ce2345babd5ecac090a44a9"} Jul 22 10:03:51.828988 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:51.828928 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf" Jul 22 10:03:51.830877 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:51.830860 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7vwf\" (UniqueName: \"kubernetes.io/projected/54d27d0d-ef7b-4e41-b3be-e67394ecd43f-kube-api-access-x7vwf\") pod \"54d27d0d-ef7b-4e41-b3be-e67394ecd43f\" (UID: \"54d27d0d-ef7b-4e41-b3be-e67394ecd43f\") " Jul 22 10:03:51.830971 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:51.830948 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d27d0d-ef7b-4e41-b3be-e67394ecd43f-openshift-service-ca\") pod \"54d27d0d-ef7b-4e41-b3be-e67394ecd43f\" (UID: \"54d27d0d-ef7b-4e41-b3be-e67394ecd43f\") " Jul 22 10:03:51.830971 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:51.830965 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54d27d0d-ef7b-4e41-b3be-e67394ecd43f-maas-api-tls\") pod \"54d27d0d-ef7b-4e41-b3be-e67394ecd43f\" (UID: \"54d27d0d-ef7b-4e41-b3be-e67394ecd43f\") " Jul 22 10:03:51.831262 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:51.831241 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54d27d0d-ef7b-4e41-b3be-e67394ecd43f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "54d27d0d-ef7b-4e41-b3be-e67394ecd43f" (UID: "54d27d0d-ef7b-4e41-b3be-e67394ecd43f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 10:03:51.832844 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:51.832816 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d27d0d-ef7b-4e41-b3be-e67394ecd43f-kube-api-access-x7vwf" (OuterVolumeSpecName: "kube-api-access-x7vwf") pod "54d27d0d-ef7b-4e41-b3be-e67394ecd43f" (UID: "54d27d0d-ef7b-4e41-b3be-e67394ecd43f"). InnerVolumeSpecName "kube-api-access-x7vwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 10:03:51.832952 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:51.832845 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54d27d0d-ef7b-4e41-b3be-e67394ecd43f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "54d27d0d-ef7b-4e41-b3be-e67394ecd43f" (UID: "54d27d0d-ef7b-4e41-b3be-e67394ecd43f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 10:03:51.931797 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:51.931770 2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d27d0d-ef7b-4e41-b3be-e67394ecd43f-openshift-service-ca\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:03:51.931797 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:51.931793 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54d27d0d-ef7b-4e41-b3be-e67394ecd43f-maas-api-tls\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:03:51.931797 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:51.931803 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7vwf\" (UniqueName: \"kubernetes.io/projected/54d27d0d-ef7b-4e41-b3be-e67394ecd43f-kube-api-access-x7vwf\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:03:52.742686 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:52.742598 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf" event={"ID":"54d27d0d-ef7b-4e41-b3be-e67394ecd43f","Type":"ContainerDied","Data":"64d769ad502fa27a0353b7aeb85039eda83f04b2d9588753048a3d573f3d1ea7"} Jul 22 10:03:52.742686 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:52.742655 2549 scope.go:117] "RemoveContainer" containerID="3566061588efe3f033b1f581ba9c0258d441e5726ce2345babd5ecac090a44a9" Jul 22 10:03:52.743132 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:52.742615 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf" Jul 22 10:03:52.759747 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:52.759724 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf"] Jul 22 10:03:52.765658 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:52.765633 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-88ceff-76dc5fc4cd-4xrjf"] Jul 22 10:03:54.339392 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:54.339361 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54d27d0d-ef7b-4e41-b3be-e67394ecd43f" path="/var/lib/kubelet/pods/54d27d0d-ef7b-4e41-b3be-e67394ecd43f/volumes" Jul 22 10:03:56.191900 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:56.191856 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:03:56.580553 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:03:56.580518 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:04:02.650918 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:02.650883 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:04:02.750891 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:02.750856 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:04:02.914530 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:02.914444 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt"] Jul 22 10:04:02.914698 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:02.914674 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" podUID="0e99b786-a54d-45bd-830e-d10b40c7b40b" containerName="istio-proxy" containerID="cri-o://73c03cea58704809bfec14b7830b9a471126b44c88006e600b049568f6c7cb5b" gracePeriod=30 Jul 22 10:04:02.931263 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:02.931236 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:04:03.177498 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:03.177414 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:04:06.138944 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:06.138905 2549 patch_prober.go:28] interesting pod/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 10:04:06.139322 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:06.138961 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" podUID="0e99b786-a54d-45bd-830e-d10b40c7b40b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 10:04:08.173974 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.173951 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 10:04:08.244038 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.243961 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-envoy\") pod \"0e99b786-a54d-45bd-830e-d10b40c7b40b\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " Jul 22 10:04:08.244038 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.244005 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-podinfo\") pod \"0e99b786-a54d-45bd-830e-d10b40c7b40b\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " Jul 22 10:04:08.244038 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.244027 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-data\") pod \"0e99b786-a54d-45bd-830e-d10b40c7b40b\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " Jul 22 10:04:08.244335 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.244044 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e99b786-a54d-45bd-830e-d10b40c7b40b-istiod-ca-cert\") pod \"0e99b786-a54d-45bd-830e-d10b40c7b40b\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " Jul 22 10:04:08.244335 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.244095 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-credential-socket\") pod \"0e99b786-a54d-45bd-830e-d10b40c7b40b\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " Jul 22 10:04:08.244335 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.244110 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfbfd\" (UniqueName: \"kubernetes.io/projected/0e99b786-a54d-45bd-830e-d10b40c7b40b-kube-api-access-hfbfd\") pod \"0e99b786-a54d-45bd-830e-d10b40c7b40b\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " Jul 22 10:04:08.244335 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.244129 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-workload-certs\") pod \"0e99b786-a54d-45bd-830e-d10b40c7b40b\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " Jul 22 10:04:08.244335 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.244144 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-token\") pod \"0e99b786-a54d-45bd-830e-d10b40c7b40b\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " Jul 22 10:04:08.244335 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.244174 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-workload-socket\") pod \"0e99b786-a54d-45bd-830e-d10b40c7b40b\" (UID: \"0e99b786-a54d-45bd-830e-d10b40c7b40b\") " Jul 22 10:04:08.244682 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.244463 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0e99b786-a54d-45bd-830e-d10b40c7b40b" (UID: "0e99b786-a54d-45bd-830e-d10b40c7b40b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:04:08.244682 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.244602 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0e99b786-a54d-45bd-830e-d10b40c7b40b" (UID: "0e99b786-a54d-45bd-830e-d10b40c7b40b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:04:08.244682 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.244628 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0e99b786-a54d-45bd-830e-d10b40c7b40b" (UID: "0e99b786-a54d-45bd-830e-d10b40c7b40b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:04:08.244845 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.244763 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e99b786-a54d-45bd-830e-d10b40c7b40b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0e99b786-a54d-45bd-830e-d10b40c7b40b" (UID: "0e99b786-a54d-45bd-830e-d10b40c7b40b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 10:04:08.244912 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.244885 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-data" (OuterVolumeSpecName: "istio-data") pod "0e99b786-a54d-45bd-830e-d10b40c7b40b" (UID: "0e99b786-a54d-45bd-830e-d10b40c7b40b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:04:08.246339 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.246298 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e99b786-a54d-45bd-830e-d10b40c7b40b-kube-api-access-hfbfd" (OuterVolumeSpecName: "kube-api-access-hfbfd") pod "0e99b786-a54d-45bd-830e-d10b40c7b40b" (UID: "0e99b786-a54d-45bd-830e-d10b40c7b40b"). InnerVolumeSpecName "kube-api-access-hfbfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 10:04:08.246480 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.246464 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-token" (OuterVolumeSpecName: "istio-token") pod "0e99b786-a54d-45bd-830e-d10b40c7b40b" (UID: "0e99b786-a54d-45bd-830e-d10b40c7b40b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 10:04:08.246638 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.246612 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0e99b786-a54d-45bd-830e-d10b40c7b40b" (UID: "0e99b786-a54d-45bd-830e-d10b40c7b40b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 10:04:08.246638 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.246616 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0e99b786-a54d-45bd-830e-d10b40c7b40b" (UID: "0e99b786-a54d-45bd-830e-d10b40c7b40b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:04:08.345214 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.345180 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-podinfo\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:04:08.345214 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.345215 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-data\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:04:08.345509 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.345229 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e99b786-a54d-45bd-830e-d10b40c7b40b-istiod-ca-cert\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:04:08.345509 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.345242 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-credential-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:04:08.345509 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.345258 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfbfd\" (UniqueName: \"kubernetes.io/projected/0e99b786-a54d-45bd-830e-d10b40c7b40b-kube-api-access-hfbfd\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:04:08.345509 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.345273 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-workload-certs\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:04:08.345509 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.345286 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-token\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:04:08.345509 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.345333 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-workload-socket\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:04:08.345509 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.345350 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e99b786-a54d-45bd-830e-d10b40c7b40b-istio-envoy\") on node \"ip-10-0-138-138.ec2.internal\" DevicePath \"\"" Jul 22 10:04:08.822207 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.822171 2549 generic.go:356] "Generic (PLEG): container finished" podID="0e99b786-a54d-45bd-830e-d10b40c7b40b" containerID="73c03cea58704809bfec14b7830b9a471126b44c88006e600b049568f6c7cb5b" exitCode=0 Jul 22 10:04:08.822390 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.822235 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" Jul 22 10:04:08.822390 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.822252 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" event={"ID":"0e99b786-a54d-45bd-830e-d10b40c7b40b","Type":"ContainerDied","Data":"73c03cea58704809bfec14b7830b9a471126b44c88006e600b049568f6c7cb5b"} Jul 22 10:04:08.822390 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.822321 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt" event={"ID":"0e99b786-a54d-45bd-830e-d10b40c7b40b","Type":"ContainerDied","Data":"8b475abf6a79efff5823c4fc60a5d4bc8c355b3ee54ae4077a6a229559437589"} Jul 22 10:04:08.822390 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.822347 2549 scope.go:117] "RemoveContainer" containerID="73c03cea58704809bfec14b7830b9a471126b44c88006e600b049568f6c7cb5b" Jul 22 10:04:08.830810 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.830792 2549 scope.go:117] "RemoveContainer" containerID="4bc4e58cc4d8f8d6a8d82f2cfdf16ca14aea1bbe2434dc14269776ff3436079a" Jul 22 10:04:08.838274 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.838258 2549 scope.go:117] "RemoveContainer" containerID="73c03cea58704809bfec14b7830b9a471126b44c88006e600b049568f6c7cb5b" Jul 22 10:04:08.838545 ip-10-0-138-138 kubenswrapper[2549]: E0722 10:04:08.838524 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73c03cea58704809bfec14b7830b9a471126b44c88006e600b049568f6c7cb5b\": container with ID starting with 73c03cea58704809bfec14b7830b9a471126b44c88006e600b049568f6c7cb5b not found: ID does not exist" containerID="73c03cea58704809bfec14b7830b9a471126b44c88006e600b049568f6c7cb5b" Jul 22 10:04:08.838608 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.838553 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c03cea58704809bfec14b7830b9a471126b44c88006e600b049568f6c7cb5b"} err="failed to get container status \"73c03cea58704809bfec14b7830b9a471126b44c88006e600b049568f6c7cb5b\": rpc error: code = NotFound desc = could not find container \"73c03cea58704809bfec14b7830b9a471126b44c88006e600b049568f6c7cb5b\": container with ID starting with 73c03cea58704809bfec14b7830b9a471126b44c88006e600b049568f6c7cb5b not found: ID does not exist" Jul 22 10:04:08.838608 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.838569 2549 scope.go:117] "RemoveContainer" containerID="4bc4e58cc4d8f8d6a8d82f2cfdf16ca14aea1bbe2434dc14269776ff3436079a" Jul 22 10:04:08.838825 ip-10-0-138-138 kubenswrapper[2549]: E0722 10:04:08.838807 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bc4e58cc4d8f8d6a8d82f2cfdf16ca14aea1bbe2434dc14269776ff3436079a\": container with ID starting with 4bc4e58cc4d8f8d6a8d82f2cfdf16ca14aea1bbe2434dc14269776ff3436079a not found: ID does not exist" containerID="4bc4e58cc4d8f8d6a8d82f2cfdf16ca14aea1bbe2434dc14269776ff3436079a" Jul 22 10:04:08.838868 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.838831 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bc4e58cc4d8f8d6a8d82f2cfdf16ca14aea1bbe2434dc14269776ff3436079a"} err="failed to get container status \"4bc4e58cc4d8f8d6a8d82f2cfdf16ca14aea1bbe2434dc14269776ff3436079a\": rpc error: code = NotFound desc = could not find container \"4bc4e58cc4d8f8d6a8d82f2cfdf16ca14aea1bbe2434dc14269776ff3436079a\": container with ID starting with 4bc4e58cc4d8f8d6a8d82f2cfdf16ca14aea1bbe2434dc14269776ff3436079a not found: ID does not exist" Jul 22 10:04:08.842112 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.842087 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt"] Jul 22 10:04:08.848864 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:08.848842 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-88ceff-openshift-default-d6fdff57f-ff7nt"] Jul 22 10:04:10.032918 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:10.032895 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67797c6b-l7dw9_29755782-61fa-4b59-9063-3b06f072620b/copy-manifests/0.log" Jul 22 10:04:10.039461 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:10.039440 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67797c6b-l7dw9_29755782-61fa-4b59-9063-3b06f072620b/manager/0.log" Jul 22 10:04:10.340099 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:10.340020 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e99b786-a54d-45bd-830e-d10b40c7b40b" path="/var/lib/kubelet/pods/0e99b786-a54d-45bd-830e-d10b40c7b40b/volumes" Jul 22 10:04:10.541347 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:10.541318 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-857c48d78c-4gkbw_602dc950-1b12-4399-8459-939eae98762c/maas-api/0.log" Jul 22 10:04:10.654689 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:10.654611 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29745210-d7b6r_70bc50eb-8892-403b-9efe-f89f89dd6341/cleanup/0.log" Jul 22 10:04:10.771380 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:10.771344 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29745225-txqdx_e811da67-1c91-42a2-9a49-725d8a247b3c/cleanup/0.log" Jul 22 10:04:11.835536 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:11.835461 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb_481aaf1f-a26a-41d3-b266-e439c874e042/util/0.log" Jul 22 10:04:11.844551 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:11.844532 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb_481aaf1f-a26a-41d3-b266-e439c874e042/pull/0.log" Jul 22 10:04:11.850395 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:11.850378 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb_481aaf1f-a26a-41d3-b266-e439c874e042/extract/0.log" Jul 22 10:04:11.964781 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:11.964752 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh_e832e7ea-798d-488b-99e7-729ae596b1db/extract/0.log" Jul 22 10:04:11.971337 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:11.971298 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh_e832e7ea-798d-488b-99e7-729ae596b1db/util/0.log" Jul 22 10:04:11.977824 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:11.977801 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh_e832e7ea-798d-488b-99e7-729ae596b1db/pull/0.log" Jul 22 10:04:12.094818 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:12.094755 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb_5f2bc8a1-a99e-4238-9f60-24098f97e5d9/util/0.log" Jul 22 10:04:12.101870 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:12.101830 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb_5f2bc8a1-a99e-4238-9f60-24098f97e5d9/pull/0.log" Jul 22 10:04:12.108387 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:12.108359 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb_5f2bc8a1-a99e-4238-9f60-24098f97e5d9/extract/0.log" Jul 22 10:04:12.230706 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:12.230684 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k_7f7434ca-9a9b-40d4-8219-d03a2e4a05f1/pull/0.log" Jul 22 10:04:12.237057 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:12.237038 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k_7f7434ca-9a9b-40d4-8219-d03a2e4a05f1/extract/0.log" Jul 22 10:04:12.243585 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:12.243567 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k_7f7434ca-9a9b-40d4-8219-d03a2e4a05f1/util/0.log" Jul 22 10:04:13.673216 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:13.673186 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-ggn8d_fc9e5acc-1a2a-4dd5-8cc5-800592c516bf/discovery/0.log" Jul 22 10:04:14.029662 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:14.029635 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-2tstv_f19d7a4f-3cb5-4652-b022-8c660e2f7c2f/istio-proxy/1.log" Jul 22 10:04:14.278893 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:14.278861 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-vjmpp_fccc33db-070e-4ab7-a476-8882378a770b/payload-processing/0.log" Jul 22 10:04:14.877692 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:14.877666 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-2gnf5_1cdd9b07-f921-4eca-b431-69039ac89c89/main/0.log" Jul 22 10:04:20.393613 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:20.393579 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:04:20.496648 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:20.496618 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" limit=1024 Jul 22 10:04:23.266339 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:23.266287 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-49l4x_a5680bfb-a669-4640-876e-84a6322b2454/global-pull-secret-syncer/0.log" Jul 22 10:04:23.411525 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:23.411494 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7lqtp_f757fe69-cbfe-4254-85cf-07a10840f85b/konnectivity-agent/0.log" Jul 22 10:04:23.531094 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:23.531073 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-138.ec2.internal_22c71e61550bcbc1af7cc7ad68e31b2b/haproxy/0.log" Jul 22 10:04:28.137900 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:28.137873 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb_481aaf1f-a26a-41d3-b266-e439c874e042/extract/0.log" Jul 22 10:04:28.170221 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:28.170196 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb_481aaf1f-a26a-41d3-b266-e439c874e042/util/0.log" Jul 22 10:04:28.196960 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:28.196936 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pmhpb_481aaf1f-a26a-41d3-b266-e439c874e042/pull/0.log" Jul 22 10:04:28.238637 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:28.238608 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh_e832e7ea-798d-488b-99e7-729ae596b1db/extract/0.log" Jul 22 10:04:28.262788 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:28.262760 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh_e832e7ea-798d-488b-99e7-729ae596b1db/util/0.log" Jul 22 10:04:28.286331 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:28.286280 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08svqh_e832e7ea-798d-488b-99e7-729ae596b1db/pull/0.log" Jul 22 10:04:28.318532 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:28.318499 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb_5f2bc8a1-a99e-4238-9f60-24098f97e5d9/extract/0.log" Jul 22 10:04:28.344084 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:28.344059 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb_5f2bc8a1-a99e-4238-9f60-24098f97e5d9/util/0.log" Jul 22 10:04:28.380618 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:28.380588 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kn2qb_5f2bc8a1-a99e-4238-9f60-24098f97e5d9/pull/0.log" Jul 22 10:04:28.412711 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:28.412637 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k_7f7434ca-9a9b-40d4-8219-d03a2e4a05f1/extract/0.log" Jul 22 10:04:28.443385 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:28.443356 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k_7f7434ca-9a9b-40d4-8219-d03a2e4a05f1/util/0.log" Jul 22 10:04:28.470276 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:28.470253 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1x766k_7f7434ca-9a9b-40d4-8219-d03a2e4a05f1/pull/0.log" Jul 22 10:04:31.440260 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:31.440232 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-njt9w_79573ff8-91b8-42b2-aefd-3c699582c1b9/node-exporter/0.log" Jul 22 10:04:31.464008 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:31.463982 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-njt9w_79573ff8-91b8-42b2-aefd-3c699582c1b9/kube-rbac-proxy/0.log" Jul 22 10:04:31.488140 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:31.488115 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-njt9w_79573ff8-91b8-42b2-aefd-3c699582c1b9/init-textfile/0.log" Jul 22 10:04:33.234464 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.234430 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df"] Jul 22 10:04:33.234892 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.234818 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54d27d0d-ef7b-4e41-b3be-e67394ecd43f" containerName="maas-api" Jul 22 10:04:33.234892 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.234833 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="54d27d0d-ef7b-4e41-b3be-e67394ecd43f" containerName="maas-api" Jul 22 10:04:33.234892 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.234849 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b6771b4-2f19-494c-b3cc-fff89d187b2b" containerName="istio-proxy" Jul 22 10:04:33.234892 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.234855 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b6771b4-2f19-494c-b3cc-fff89d187b2b" containerName="istio-proxy" Jul 22 10:04:33.234892 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.234865 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b6771b4-2f19-494c-b3cc-fff89d187b2b" containerName="istio-proxy" Jul 22 10:04:33.234892 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.234873 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b6771b4-2f19-494c-b3cc-fff89d187b2b" containerName="istio-proxy" Jul 22 10:04:33.234892 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.234889 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c980835c-4818-4404-b928-afbda5084a6b" containerName="maas-api" Jul 22 10:04:33.234892 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.234893 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="c980835c-4818-4404-b928-afbda5084a6b" containerName="maas-api" Jul 22 10:04:33.235141 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.234900 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e99b786-a54d-45bd-830e-d10b40c7b40b" containerName="istio-proxy" Jul 22 10:04:33.235141 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.234906 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e99b786-a54d-45bd-830e-d10b40c7b40b" containerName="istio-proxy" Jul 22 10:04:33.235141 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.234911 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e99b786-a54d-45bd-830e-d10b40c7b40b" containerName="istio-proxy" Jul 22 10:04:33.235141 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.234916 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e99b786-a54d-45bd-830e-d10b40c7b40b" containerName="istio-proxy" Jul 22 10:04:33.235141 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.234967 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e99b786-a54d-45bd-830e-d10b40c7b40b" containerName="istio-proxy" Jul 22 10:04:33.235141 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.234975 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="54d27d0d-ef7b-4e41-b3be-e67394ecd43f" containerName="maas-api" Jul 22 10:04:33.235141 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.234980 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b6771b4-2f19-494c-b3cc-fff89d187b2b" containerName="istio-proxy" Jul 22 10:04:33.235141 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.234986 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="c980835c-4818-4404-b928-afbda5084a6b" containerName="maas-api" Jul 22 10:04:33.235141 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.234991 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b6771b4-2f19-494c-b3cc-fff89d187b2b" containerName="istio-proxy" Jul 22 10:04:33.237951 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.237935 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df" Jul 22 10:04:33.240209 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.240187 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nzwpt\"/\"openshift-service-ca.crt\"" Jul 22 10:04:33.240489 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.240468 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nzwpt\"/\"kube-root-ca.crt\"" Jul 22 10:04:33.241296 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.241277 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nzwpt\"/\"default-dockercfg-54vb5\"" Jul 22 10:04:33.253028 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.253006 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df"] Jul 22 10:04:33.300995 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.300975 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-jwgdc_adc69b3e-14e2-4d1d-9a65-7a89d88190ff/networking-console-plugin/0.log" Jul 22 10:04:33.360712 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.360685 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1baff34-25d0-4d63-b5a0-0a97704bf8fd-sys\") pod \"perf-node-gather-daemonset-mv8df\" (UID: \"b1baff34-25d0-4d63-b5a0-0a97704bf8fd\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df" Jul 22 10:04:33.360821 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.360719 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b1baff34-25d0-4d63-b5a0-0a97704bf8fd-proc\") pod \"perf-node-gather-daemonset-mv8df\" (UID: \"b1baff34-25d0-4d63-b5a0-0a97704bf8fd\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df" Jul 22 10:04:33.360821 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.360750 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b1baff34-25d0-4d63-b5a0-0a97704bf8fd-podres\") pod \"perf-node-gather-daemonset-mv8df\" (UID: \"b1baff34-25d0-4d63-b5a0-0a97704bf8fd\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df" Jul 22 10:04:33.360914 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.360818 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm6ks\" (UniqueName: \"kubernetes.io/projected/b1baff34-25d0-4d63-b5a0-0a97704bf8fd-kube-api-access-nm6ks\") pod \"perf-node-gather-daemonset-mv8df\" (UID: \"b1baff34-25d0-4d63-b5a0-0a97704bf8fd\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df" Jul 22 10:04:33.360914 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.360853 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1baff34-25d0-4d63-b5a0-0a97704bf8fd-lib-modules\") pod \"perf-node-gather-daemonset-mv8df\" (UID: \"b1baff34-25d0-4d63-b5a0-0a97704bf8fd\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df" Jul 22 10:04:33.461985 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.461938 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1baff34-25d0-4d63-b5a0-0a97704bf8fd-sys\") pod \"perf-node-gather-daemonset-mv8df\" (UID: \"b1baff34-25d0-4d63-b5a0-0a97704bf8fd\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df" Jul 22 10:04:33.462115 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.461994 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b1baff34-25d0-4d63-b5a0-0a97704bf8fd-proc\") pod \"perf-node-gather-daemonset-mv8df\" (UID: \"b1baff34-25d0-4d63-b5a0-0a97704bf8fd\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df" Jul 22 10:04:33.462115 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.462029 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b1baff34-25d0-4d63-b5a0-0a97704bf8fd-podres\") pod \"perf-node-gather-daemonset-mv8df\" (UID: \"b1baff34-25d0-4d63-b5a0-0a97704bf8fd\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df" Jul 22 10:04:33.462115 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.462058 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1baff34-25d0-4d63-b5a0-0a97704bf8fd-sys\") pod \"perf-node-gather-daemonset-mv8df\" (UID: \"b1baff34-25d0-4d63-b5a0-0a97704bf8fd\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df" Jul 22 10:04:33.462115 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.462060 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm6ks\" (UniqueName: \"kubernetes.io/projected/b1baff34-25d0-4d63-b5a0-0a97704bf8fd-kube-api-access-nm6ks\") pod \"perf-node-gather-daemonset-mv8df\" (UID: \"b1baff34-25d0-4d63-b5a0-0a97704bf8fd\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df" Jul 22 10:04:33.462347 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.462118 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1baff34-25d0-4d63-b5a0-0a97704bf8fd-lib-modules\") pod \"perf-node-gather-daemonset-mv8df\" (UID: \"b1baff34-25d0-4d63-b5a0-0a97704bf8fd\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df" Jul 22 10:04:33.462347 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.462130 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b1baff34-25d0-4d63-b5a0-0a97704bf8fd-proc\") pod \"perf-node-gather-daemonset-mv8df\" (UID: \"b1baff34-25d0-4d63-b5a0-0a97704bf8fd\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df" Jul 22 10:04:33.462347 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.462177 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b1baff34-25d0-4d63-b5a0-0a97704bf8fd-podres\") pod \"perf-node-gather-daemonset-mv8df\" (UID: \"b1baff34-25d0-4d63-b5a0-0a97704bf8fd\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df" Jul 22 10:04:33.462347 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.462248 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1baff34-25d0-4d63-b5a0-0a97704bf8fd-lib-modules\") pod \"perf-node-gather-daemonset-mv8df\" (UID: \"b1baff34-25d0-4d63-b5a0-0a97704bf8fd\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df" Jul 22 10:04:33.470741 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.470721 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm6ks\" (UniqueName: \"kubernetes.io/projected/b1baff34-25d0-4d63-b5a0-0a97704bf8fd-kube-api-access-nm6ks\") pod \"perf-node-gather-daemonset-mv8df\" (UID: \"b1baff34-25d0-4d63-b5a0-0a97704bf8fd\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df" Jul 22 10:04:33.547790 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.547767 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df" Jul 22 10:04:33.675651 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.675631 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df"] Jul 22 10:04:33.678114 ip-10-0-138-138 kubenswrapper[2549]: W0722 10:04:33.678084 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb1baff34_25d0_4d63_b5a0_0a97704bf8fd.slice/crio-52c080d0e2b1a41cb1cb6f7d711f96010c7bb80176ceea653770a51b38e2bc74 WatchSource:0}: Error finding container 52c080d0e2b1a41cb1cb6f7d711f96010c7bb80176ceea653770a51b38e2bc74: Status 404 returned error can't find the container with id 52c080d0e2b1a41cb1cb6f7d711f96010c7bb80176ceea653770a51b38e2bc74 Jul 22 10:04:33.948860 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.948779 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df" event={"ID":"b1baff34-25d0-4d63-b5a0-0a97704bf8fd","Type":"ContainerStarted","Data":"a06203c4372cebf3f180670808cc13c360d1a5cdd1c86fc8d9ff863bb584b6b4"} Jul 22 10:04:33.948860 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.948828 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df" event={"ID":"b1baff34-25d0-4d63-b5a0-0a97704bf8fd","Type":"ContainerStarted","Data":"52c080d0e2b1a41cb1cb6f7d711f96010c7bb80176ceea653770a51b38e2bc74"} Jul 22 10:04:33.949079 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.948915 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df" Jul 22 10:04:33.968892 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:33.968840 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df" podStartSLOduration=0.968825134 podStartE2EDuration="968.825134ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 10:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 10:04:33.967095293 +0000 UTC m=+2804.430889732" watchObservedRunningTime="2026-07-22 10:04:33.968825134 +0000 UTC m=+2804.432619572" Jul 22 10:04:34.603883 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:34.603854 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-fc8pn_3dc0aeb5-0a3f-485f-bf41-6d4c083faee4/download-server/0.log" Jul 22 10:04:36.277121 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:36.277093 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k4mr8_44d91ea8-9844-4ae9-9af8-b5297d3e1639/dns/0.log" Jul 22 10:04:36.303431 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:36.303403 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k4mr8_44d91ea8-9844-4ae9-9af8-b5297d3e1639/kube-rbac-proxy/0.log" Jul 22 10:04:36.438730 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:36.438696 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-659hp_a87cdd8f-1617-4f31-85ab-6ba3294ef8be/dns-node-resolver/0.log" Jul 22 10:04:37.053086 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:37.053058 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5dc8c5f9bf-7rhc6_28c1f02e-5341-49ff-9bd0-8a188443891e/registry/0.log" Jul 22 10:04:37.102397 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:37.102374 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rn8j7_beb97b8f-7ed2-41fc-87b5-7a0488638e16/node-ca/0.log" Jul 22 10:04:38.225744 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:38.225710 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-ggn8d_fc9e5acc-1a2a-4dd5-8cc5-800592c516bf/discovery/0.log" Jul 22 10:04:38.313203 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:38.313173 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-2tstv_f19d7a4f-3cb5-4652-b022-8c660e2f7c2f/istio-proxy/0.log" Jul 22 10:04:38.338557 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:38.338533 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-2tstv_f19d7a4f-3cb5-4652-b022-8c660e2f7c2f/istio-proxy/1.log" Jul 22 10:04:38.400918 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:38.400890 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-vjmpp_fccc33db-070e-4ab7-a476-8882378a770b/payload-processing/0.log" Jul 22 10:04:39.247300 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:39.247267 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wzwlh_f40f3b1a-129d-42b5-8b2d-99e272295571/serve-healthcheck-canary/0.log" Jul 22 10:04:39.921466 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:39.921440 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xrlw2_263ef12e-19a9-47c9-adc5-e8b0ec511533/kube-rbac-proxy/0.log" Jul 22 10:04:39.944569 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:39.944541 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xrlw2_263ef12e-19a9-47c9-adc5-e8b0ec511533/exporter/0.log" Jul 22 10:04:39.964080 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:39.964059 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-mv8df" Jul 22 10:04:39.970068 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:39.970049 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xrlw2_263ef12e-19a9-47c9-adc5-e8b0ec511533/extractor/0.log" Jul 22 10:04:42.717189 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:42.717160 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67797c6b-l7dw9_29755782-61fa-4b59-9063-3b06f072620b/manager/0.log" Jul 22 10:04:42.744835 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:42.744806 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67797c6b-l7dw9_29755782-61fa-4b59-9063-3b06f072620b/copy-manifests/0.log" Jul 22 10:04:44.179279 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:44.179246 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7d9cd798d9-f4jnw_1467962b-93b6-45dc-bf90-3715e9c078c4/manager/0.log" Jul 22 10:04:44.201212 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:44.201161 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-ggn8d_fc9e5acc-1a2a-4dd5-8cc5-800592c516bf/discovery/0.log" Jul 22 10:04:44.250898 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:44.250870 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-2tstv_f19d7a4f-3cb5-4652-b022-8c660e2f7c2f/istio-proxy/0.log" Jul 22 10:04:44.277801 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:44.277766 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-2tstv_f19d7a4f-3cb5-4652-b022-8c660e2f7c2f/istio-proxy/1.log" Jul 22 10:04:44.321019 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:44.320989 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-vjmpp_fccc33db-070e-4ab7-a476-8882378a770b/payload-processing/0.log" Jul 22 10:04:45.653118 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:45.653080 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-ggn8d_fc9e5acc-1a2a-4dd5-8cc5-800592c516bf/discovery/0.log" Jul 22 10:04:45.713985 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:45.713946 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-2tstv_f19d7a4f-3cb5-4652-b022-8c660e2f7c2f/istio-proxy/0.log" Jul 22 10:04:45.739948 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:45.739920 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-2tstv_f19d7a4f-3cb5-4652-b022-8c660e2f7c2f/istio-proxy/1.log" Jul 22 10:04:45.784698 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:45.784666 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-vjmpp_fccc33db-070e-4ab7-a476-8882378a770b/payload-processing/0.log" Jul 22 10:04:49.286780 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:49.286746 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-5982h_19a238da-30ed-4a74-9f90-2a32f8076615/migrator/0.log" Jul 22 10:04:49.315902 ip-10-0-138-138 kubenswrapper[2549]: I0722 10:04:49.315872 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-5982h_19a238da-30ed-4a74-9f90-2a32f8076615/graceful-termination/0.log"